Proceedings of the Annual Meeting of the Cognitive Science Society

Identifying the visual referent of a spoken word – that a partic-ular insect is referred to by the word “bee” – requires both theability to process and integrate multimodal input and the abil-ity to reason under uncertainty. How do these tasks interactwith one another? We introduce a task that allows us to ex-amine how adults identify words under joint uncertainty in theauditory and visual modalities. We propose an ideal observermodel of the task which provides an optimal baseline. Modelpredictions are tested in two experiments where word recogni-tion is made under two kinds of uncertainty: category ambigu-ity and distorting noise. In both cases, the ideal observer modelexplains much of the variance in human judgments. But whenone modality had noise added to it, human perceivers system-atically preferred the unperturbed modality to a greater extentthan the ideal observer model did.