© 2015 American Academy of Neurology. Objective: The objective of this study was to test whether effects of β-amyloid (Aβ) pathology on episodic memory were mediated by metabolism and gray matter volume in the early stages of Alzheimer disease. Methods: This was a prospective cohort study. We measured baseline Aβ (using florbetapir-PET), brain function (using fluorodeoxyglucose-PET), and brain structure (using MRI). A mediation analysis was performed to test whether statistical effects of Aβ positivity on cross-sectional and longitudinal episodic memory were mediated by hypometabolism or regional gray matter volume in cognitively healthy controls (CN, n = 280) and mild cognitive impairment (MCI, n = 463). Results: Lower memory scores were associated with Aβ positivity (CN, mildly; MCI, strongly), smaller gray matter volumes (CN, few regions, including hippocampus; MCI, widespread), and hypometabolism. Smaller volumes and hypometabolism mediated effects of Aβ in MCI but not in CN. The strongest individual regions mediated up to approximately 25%. A combination of brain structure and function mediated up to approximately 40%. In several regions, gray matter atrophy and hypometabolism predicted episodic memory without being associated (at p < 0.05) with Aβ positivity. Conclusions: Changes in brain structure and function appear to be, in part, downstream events from Aβ pathology, ultimately resulting in episodic memory deficits. However, Aβ pathology is also strongly related to memory deficits through mechanisms that are not quantified by these imaging measurements, and episodic memory decline is partly caused by Alzheimer disease-like brain changes independently of Aβ pathology.