Benefits to lexical access are provided by acoustically-cued speaker characteristics (such as gender and age), but little work has investigated these effects in meaning-based tasks. Word recognition is affected both by a word’s base-level activation and by associative spread of activation among words, and is correlated with speed of lexical access. In a free association task and a semantic priming task, we find off-line and on-line evidence of speaker-specific relationships between words. Our results suggest the need to extend existing models of spoken word recognition to include interactions between linguistic information and social information that is cued by variation in speech.