How are our conceptual representations organized, and how does this semantic structure differ in individuals or groups? We included 36 faces from the Chicago Face Database (Ma et al., 2015) varying in race, gender, and facial expression in a triplet judgement task, where participants were showed a target face and asked which of two other faces is most similar to the target face expression. Each of our 71 participants completed 1,012 triplet judgements. Using their judgements, we fit individualized embeddings capturing distances between faces in semantic space, resulting in 71 embeddings. We calculated the Procrustes similarity between all pairs of embeddings and clustered similarities, revealing four clusters of respondents. Each cluster organized faces differently, indicating group-wide differences in semantic organization of expression. At the individual level, we find that each participant’s responses for held-out items are better predicted by their own embedding than by other participants’ embeddings, p < .00001.

Our interaction with others largely hinges on how we semantically organize the social world. The organization of such conceptual information is not static—as we age, our experiences and ever-changing anatomy alter how we represent and arrange semantic information. How does semantic distance between concepts affect this organization, particularly for those with pathological deficits in semantic knowledge? Using triplet judgment responses collected from healthy participants, we compute an ordinal similarity embedding for a set of social words and images that vary in the dimensions of age and gender. We compare semantic distances between items in the space to patterns of error in a word-picture matching task performed by patients with semantic dementia (SD). Error patterns reveal that SD patients retain gender information more robustly than age information, and that age-related errors are a function of linear distance in age from a concept word. The distances between probed and exemplar items in the resulting conceptual map reflect error patterns in SD patient responses such that items semantically closer to a probed concept—in gender category or in linear age—are more likely to be erroneously chosen by patients in a word-picture matching task. To our knowledge, this is the first triplet embedding work to embed representations of words and images in a unified space, and to use this space to explain patterns of behavior in patients with impaired social semantic cognition.