Proceedings of the Annual Meeting of the Cognitive Science Society

Brain-based features of meaning (sensory-motor features: sound, color, manipulation, motion, and shape) are usedto compare two popular models of adjective-noun semantic composition: element-wise vector addition and multiplication. Alarge literature (e.g. Fernandino et al., 2015) suggests that perceptual systems contain information that can be extracted usingneural decoding (e.g. Anderson, Murphy & Poesio, 2014). Using Amazon’s Mechanical Turk, participants rated how mucheach of the words and phrases (made of all combinations of the selected adjectives and nouns) evoked the features. Bothmultiplication and addition surpass chance at matching the correct phrase, but addition outperformed multiplication (addition =7.6/60, multiplication = 13.4/60). Addition allows the adjective to weight the important sensory-motor attributes for the noun.Based on these behavioral results, we predict, and will test in upcoming work, that addition will also be successful when usingbrain activity (from fMRI) as the representations of the adjectives, nouns, and phrases.