This work investigates the role that pore structure plays in colloid retention across scales with a novel methodology based on image analysis. Experiments were designed to quantify–with robust statistics–the contribution from commonly proposed retention sites toward colloid immobilization. Specific retention sites include solid-water interface, air-water interface, air-water-solid triple point, grain-to-grain contacts, and thin films. Variable conditions for pore-water content, velocity, and chemistry were tested in a model glass bead porous medium with silver microspheres. Concentration signals from effluent breakthrough and spatial profiles of retained particles from micro X-ray Computed Tomography were used to compute mass balances and enumerate pore-scale regions of interest in three dimensions. At the Darcy-scale, retained colloids follow non-monotonic deposition profiles, which implicates effects from flow-stagnation zones. The spatial distribution of immobilized colloids along the porous medium depth was analyzed by retention site, revealing depth-independent partitioning of colloids. At the pore-scale, dominance and overall saturation of all retention sites considered indicated that the solid-water interface and wedge-shaped regions associated with flow-stagnation (grain-to-grain contacts in saturated and air-water-solid triple points in unsaturated conditions) are the greatest contributors toward retention under the tested conditions. At the interface-scale, xDLVO energy profiles were in agreement with pore-scale observations. Our calculations suggest favorable interactions for colloids and solid-water interfaces and for weak flocculation (e.g., at flow-stagnation zones), but unfavorable interactions between colloids and air-water interfaces. Overall, we demonstrate that pore-structure plays a critical role in colloid immobilization and that Darcy-, pore- and interface-scales are consistent when the pore structure is taken into account.