How people reach consensus in social networks with locally distributed interactions is relevant to understanding collective group decision-making and problem-solving. However, while the importance of theory of mind in consensus problems has been hypothesized, little work has been done to test it systematically. We present both computational modeling and behavioral experiments designed to test the impact of theory of mind on individual choices within such consensus networks. We test 2,108 computational models informed by theoretical work on a graph-coloring consensus task to compare models using theory of mind to other behavioral parameters. We then use behavioral responses from 107 participants in a similar task to evaluate support for theory of mind in consensus formation. We find that the computational model that best accounts for prior behavioral data uses theory of mind, and our behavioral results likewise support use of theory of mind over other potential decision-making models.

We make aesthetic judgments on a daily basis. While we think of these judgments as highly personal, they are often shaped by social context. This poses a computational problem: how do we combine social information and our individual judgments to produce a single evaluation? In this study, we examine social influence on aesthetic evaluations in online transmission chain experiments. We test not only the effect of social information, but also variation in effect depending on how information is presented--echoing the variety of interfaces we encounter in naturalistic cases. We find that social information significantly affects ratings across interfaces. Moreover, people tend to rely more heavily on their own judgment than on social information, compared to an ideally noise-reducing model for combining multiple signals. These results offer detailed insight into the formation of aesthetic judgment and suggest the need for extended investigation into social influence on subjective judgments more broadly.

Similarity judgments provide a well-established method for accessing mental representations, with applications in psychology, neuroscience and machine learning. However, collecting similarity judgments can be prohibitively expensive for naturalistic datasets as the number of comparisons grows quadratically in the number of stimuli. One way to tackle this problem is to construct approximation procedures that rely on more accessible proxies for predicting similarity. Here we leverage recent advances in language models and online recruitment, proposing an efficient domain-general procedure for predicting human similarity judgments based on text descriptions. Intuitively, similar stimuli are likely to evoke similar descriptions, allowing us to use description similarity to predict pairwise similarity judgments. Crucially, the number of descriptions required grows only linearly with the number of stimuli, drastically reducing the amount of data required. We test this procedure on six datasets of naturalistic images and show that our models outperform previous approaches based on visual information.

Understanding the extent to which the perceptual world can be recovered from language is a fundamental problem in cognitive science. We reformulate this problem as that of distilling psychophysical information from text and show how this can be done by combining large language models (LLMs) with a classic psychophysical method based on similarity judgments. Specifically, we use the prompt completion functionality of GPT3, a state-of-the-art LLM, to elicit similarity scores between stimuli and then apply multidimensional scaling to uncover their underlying psychological space. We test our approach on six perceptual domains and show that the elicited judgments strongly correlate with human data and successfully recover well-known psychophysical structures such as the color wheel and pitch spiral. We also explore meaningful divergences between LLM and human representations. Our work showcases how combining state-of-the-art machine models with well-known cognitive paradigms can shed new light on fundamental questions in perception and language research.

The speech-to-song illusion is a robust psychological phenomenon whereby a spoken sentence sounds increasingly more musical as it is repeated. Despite decades of research, a complete formal account of this transformation is still lacking, and some of its nuanced characteristics, namely, that certain phrases appear to transform while others do not, is not well understood. Here we provide a formal account of this phenomenon, by recasting it as a statistical inference whereby a rational agent attempts to decide whether a sequence of utterances is more likely to have been produced in a song or speech. Using this approach and analyzing song and speech corpora, we further introduce a novel prose-to-lyrics illusion that is purely text-based. In this illusion, simply duplicating written sentences makes them appear more like song lyrics. We provide robust evidence for this new illusion in both human participants and large language models.

How does globalization impact the interaction between perception and language? Building on Berlin and Kay's foundational study of color naming, we recruited 2,280 online participants speaking 22 different languages. We show that color naming maps differ structurally across languages, even among internet users living in (mostly) industrial societies. We use Large Language Models (LLMs) to simulate the limits of globalization by reproducing the naming task with a highly multilingual artificial agent with access to global digital information. We show that while the LLM has access to all languages, it has language-specific color representations and the number of color terms is correlated across humans and LLMs. However, LLMs use more color terms than humans, indicating differences in the representation. These results suggest that globalization has not removed cultural distinctions in color concepts, as language continues to be a key factor in the diversity of perception and meaning.

Humans extract useful abstractions of the world from noisy sensory data. Serial reproduction allows us to study how people construe the world through a paradigm similar to the game of telephone, where one person observes a stimulus and reproduces it for the next to form a chain of reproductions. Past serial reproduction experiments typically employ a single sensory modality, but humans often communicate abstractions of the world to each other through language. To investigate the effect language on the formation of abstractions, we implement a novel multimodal serial reproduction framework by asking people who receive a visual stimulus to reproduce it in a linguistic format, and vice versa. We ran unimodal and multimodal chains with both humans and GPT-4 and find that adding language as a modality has a larger effect on human reproductions than GPT-4's. This suggests human visual and linguistic representations are more dissociable than those of GPT-4.

Conducting experiments with diverse participants in their native languages can uncover insights into culture, cognition, and language that may not be revealed otherwise. However, conducting these experiments online makes it difficult to validate self-reported language proficiency. Furthermore, existing proficiency tests are small and cover only a few languages. We present an automated pipeline to generate vocabulary tests using text from Wikipedia. Our pipeline samples rare nouns and creates pseudowords with the same low-level statistics. Six behavioral experiments (N=236) in six countries and eight languages show that (a) our test can distinguish between native speakers of closely related languages, (b) the test is reliable (r=0.82), and (c) performance strongly correlates with existing tests (LexTale) and self-reports. We further show that test accuracy is negatively correlated with the linguistic distance between the tested and the native language. Our test, available in eight languages, can easily be extended to other languages.