As a relatively recent cultural invention in human evolution, reading is an important gateway to personal development and socioeconomic success. Despite the well documented individual differences in reading ability, its neuroanatomical correlates have not been well understood, largely due to the fact that reading is a complex skill that consists of multiple components. Using a large sample of 416 college students and 7 reading tasks, the present study successfully identified three uncorrelated components of reading ability: phonological decoding, form-sound association, and naming speed. We then tried to predict individuals' scores in these components from their gray matter volume (GMV) on a subset of participants (N = 253) with high-quality structural images, adopting a multivariate support vector regression analysis with tenfold cross-validation. Our results revealed distinct neural regions that supported different aspects of reading ability: whereas phonological decoding was associated with the GMV in the left superior parietal lobe extending to the supramarginal gyrus, form-sound association was predicted by the GMV in the hippocampus and cerebellum. Naming speed was associated with GMV in distributed brain regions in the occipital, temporal, parietal, and frontal cortices. Phonological decoding and form-sound association were uncorrelated with general cognitive abilities. However, naming speed was correlated with intelligence and processing speed, and some of the regions that were predictive of naming speed also predicted these general cognitive abilities. These results provide further insights on the cognitive and neural architecture of reading and the structural basis of individual differences in reading abilities.