Attaching semantic meaning to sensory information received from both inside and outside our bodies is a fundamental function of the human brain. The theory of Controlled Semantic Cognition (CSC) proposes that the formation of semantic knowledge relies on connections between spatially distributed modality-specific spoke-nodes, and a modality-general hub in the anterior temporal lobes (ATLs). This theory can also be applied to social semantic knowledge, though certain domain-specific spoke-nodes may make a disproportionate contribution to the understanding of social concepts. The ATLs have strong connections with spoke-node structures such as the subgenual ACC (sgACC) and the orbitofrontal cortex (OFC) that play an important role in predicting the hedonic value of stimuli. We hypothesized that in addition to the ATL semantic hub, a social semantic task would also require input from hedonic evaluation structures. We used voxel based morphometry (VBM) to examine structural brain-behavior relationships in 152 patients with neurodegeneration (Alzheimer's disease [N = 12], corticobasal syndrome (N = 18], progressive supranuclear palsy [N = 13], behavioral variant frontotemporal dementia [N = 56], and primary progressive aphasia (PPA) [N = 53]) using the Social Interaction Vocabulary Task (SIVT). This task measures the ability to correctly match a social term (e.g. "gossiping") with a visual depiction of that social interaction. As predicted, VBM showed that worse SIVT scores corresponded with volume loss in bilateral ATL semantic hub regions, but also in the sgACC, OFC, caudate and putamen (pFWE <0.05). These results support the CSC model of a hub-and-spoke organization of social semantic knowledge with the ATL as a domain-general semantic hub, and ventromedial and striatal structures as domain specific spoke-nodes. Importantly, these results suggest that correct comprehension of social semantic concepts requires emotional 'tagging' of a concept by the evaluation system, and that the social deficits observed in some neurodegenerative disease syndromes may be caused by the break-down of this mechanism.