Proceedings of the Annual Meeting of the Cognitive Science Society

The behaviors of Large Language Models (LLMs) as artificial social actors are largely underexplored, particularly in unverifiable scenarios where conventional benchmarking is often not applicable. Thus, examining their behaviors in such scenarios can help understand and improve LLMs' capabilities of simulating real-world social actors in many tasks such as LLM-empowered agents. We draw a typical unverifiable scenario--a simplified pull request scenario on \textsc{GitHub} focusing on decision-making based on Activity Overview signal--to investigate how human and LLMs behave. We introduce a method to collect, compare, and reason about human and LLMs' decisions. We reveal that there are both similarities and differences between human mind and LLMs' decisions, and proprietary LLMs generally behave more like human than open-source LLMs do. We further find that human and LLMs may rely on different information and reasoning mechanisms in decision-making. Our study thus urges more work on human and LLMs decision-making in unverifiable environments.