In a cohort of community-recruited elderly subjects with normal cognition at initial evaluation, we found that baseline fornix white matter (WM) microstructure was significantly correlated with early volumetric longitudinal tissue change across a region of interest (called fornix significant ROI, fSROI), which overlaps circuits known to be selectively vulnerable to Alzheimer's dementia pathology. Other WM and gray matter regions had much weaker or non-existent associations with longitudinal tissue change. Tissue loss in fSROI was in turn a significant factor in a survival model of cognitive decline, as was baseline fornix microstructure. These findings suggest that WM deterioration in the fornix and tissue loss in fSROI may be the early beginnings of posterior limbic circuit and default mode network degeneration. We also found that gray matter baseline volumes in the entorhinal cortex and hippocampus predicted cognitive decline in survival models. But since GM regions did not also significantly predict brain-tissue loss, our results may imply a view in which early, prodromal deterioration appears as two quasi independent processes in white and gray matter regions of the limbic circuit crucial to memory.