Evidence from animal and histological studies has indicated that accumulation of iron in the brain results in reactive gliosis that contributes to cognitive deficits. The current study extends these findings to human cognitive aging and suggests that magnetic resonance imaging (MRI) techniques like quantitative relaxometry can be used to study iron and its effects in vivo. The effects of iron on microstructure and memory performance were examined using a combination of quantitative relaxometry and multicompartment diffusion imaging in 35 young (21.06 ± 2.18 years) and 28 older (72.58 ± 6.47 years) adults, who also completed a memory task. Replicating past work, results revealed age-related increases in iron content (R2*) and diffusion, and decreases in memory performance. Independent of age group, iron content was significantly related to restricted (intracellular) diffusion in regions with low-moderate iron (hippocampus, caudate) and to all diffusion metrics in regions with moderate-high iron (putamen, globus pallidus). This pattern is consistent with different stages of iron-related gliosis, ranging from astrogliosis that may influence intracellular diffusion to microglial proliferation and increased vascular permeability that may influence all sources of diffusion. Further, hippocampal restricted diffusion was significantly related to memory performance, with a third of this effect related to iron content; consistent with the hypothesis that higher iron-related astrogliosis in the hippocampus is associated with poorer memory performance. These results demonstrate the sensitivity of MRI to iron-related gliosis and extend our understanding of its impact on cognition by showing that this relationship also explains individual differences in memory performance.