Introduction: Patients with transient ischemic attack (TIA) and minor stroke demonstrate cognitive impairment, and a four-fold risk of late-life dementia. Aim: To study the extent to which the rates of brain volume loss in TIA patients differ from healthy controls and how they are correlated with cognitive impairment. Methods: TIA or minor stroke patients were tested with a neuropsychological battery and underwent T1 weighted volumetric magnetic resonance imaging scans at fixed intervals over a 3 years period. Linear mixed effects regression models were used to compare brain atrophy rates between groups, and to determine the relationship between atrophy rates and cognitive function in TIA and minor stroke patients. Results: Whole brain atrophy rates were calculated for the TIA and minor stroke patients; n = 38 between 24 h and 18 months, and n = 68 participants between 18 and 36 months, and were compared to healthy controls. TIA and minor stroke patients demonstrated a significantly higher whole brain atrophy rate than healthy controls over a 3 years interval (p = 0.043). Diabetes (p = 0.012) independently predicted higher atrophy rate across groups. There was a relationship between higher rates of brain atrophy and processing speed (composite P = 0.047 and digit symbol coding P = 0.02), but there was no relationship with brain atrophy rates and memory or executive composite scores or individual cognitive tests for language (Boston naming, memory recall, verbal fluency or Trails A or B score). Conclusion: TIA and minor stroke patients experience a significantly higher rate of whole brain atrophy. In this cohort of TIA and minor stroke patients changes in brain volume over time precede cognitive decline.