Skip to main content
eScholarship
Open Access Publications from the University of California

How Far Does Vision Go? Interpretable Visual Statistics from Word-Linked Web Image Distributions Predict Multisensory Norms for Chinese Words

Creative Commons 'BY' version 4.0 license
Abstract

Recent work reports that large language models, including those with visual training, fail to recover the sensorimotor structure of human concepts. We argue this reflects how visual information has been operationalized, as opaque embeddings tied to single training instances, rather than a limit of vision itself. We test this by representing 593 Chinese disyllabic words through interpretable low-level visual statistics, pixel-level and mid-level features aggregated over word-linked, CLIP-filtered web images, and evaluating their predictive power for human sensorimotor norms across six modalities beyond lexical controls. We find a graded profile of cross-modal decodability. Visual norms are predicted best, followed by gustatory, olfactory, tactile, and auditory, while interoception sits at the predicted lower bound. The graded pattern, with substantial gains for non-visual modalities, indicates that prior negative results reflect a choice of representation rather than a failure of low-level visual statistics per se.