Human decision-making can be irrational, as in the case of the gambler's fallacy (GF). Converging evidence from behavioral and functional neuroimaging studies has suggested that a hyperactive cognitive system and a hypo-active affective system contribute to the false world model that generates the GF. However, the neuroanatomical basis underlying the GF remains unclear. The current study addressed this issue by collecting high-resolution magnetic resonance anatomical images from a large sample of 350 healthy Chinese adults. Univariate voxel-based morphometry (VBM) analysis suggested that the gray matter volume (GMV) in the anterior cingulate cortex (ACC) and medial temporal lobe (MTL) (two regions of the cognitive system) showed negative correlations with the degree of GF, while the GMV in the striatum and orbitofrontal cortex (OFC; two regions of the affective system) showed positive correlations. Further multivariate VBM analysis showed that the GMV in these regions could potentially predict the degree of GF. Moreover, a mediation analysis suggested that the GMV in MTL, ACC, and OFC mediated the relationships between the cognitive abilities or affective decision-making performance and the GF. Results of our study help us to understand the potential neural bases of the cognitive system's constructive role and the affective system's destructive role in decision making.