Spurious correlations were found to be an important factor explaining model performance in various NLP tasks (e.g., gender or racial artifacts), often considered to be “shortcuts” to the actual task. However, humans tend to similarly make quick (and sometimes wrong) predictions based on societal and cognitive presuppositions. In this work we address the question: can we quantify the extent to which model biases reflect human behaviour? Answering this question will help shed light on model performance and provide meaningful comparisons against humans. We approach this question through the lens of the dual-process theory for human decision-making. This theory differentiates between an automatic unconscious (and sometimes biased) “fast system” and a “slow system”, which when triggered may revisit earlier automatic reactions. We make several observations from two crowdsourcing experiments of gender bias in coreference resolution, using self- paced reading to study the “fast” system, and question answering to study the “slow” system under a constrained time setting. On real-world data humans make ∼3% more gender- biased decisions compared to models, while on synthetic data models are ∼12% more biased. We make all our of our code and data publicly available.

Cross-domain alignment refers to the task of mapping a concept from one domain to another, for example, ``If a \textit{doctor} were a \textit{color}, what color would it be?''. This seemingly peculiar task was designed to investigate how people represent concrete and abstract concepts through their mappings between categories and their reasoning processes over those mappings. In this paper, we adapt this task from cognitive science to evaluate the conceptualization and reasoning abilities of large language models (LLMs) through a behavioral study. We examine several LLMs by prompting them with a cross-domain mapping task and analyzing their responses at the population level and the individual level. Additionally, we assess the models' ability to reason about their predictions by analyzing and categorizing their explanations for these mappings. The results reveal several similarities between humans' and models' mappings and explanations, suggesting that models represent concepts similarly to humans. This similarity is evident not only at the model representation level but also in their behavior. Furthermore, the models mostly provide valid explanations and deploy reasoning paths that are similar to humans.