Proceedings of the Annual Meeting of the Cognitive Science Society

Large language models (LLMs) are increasingly used in political contexts. LLMs have been shown to exhibit biases in their knowledge representations of political topics, yet little is known about how these biases influence their reasoning about associated informal arguments. We investigated six popular LLMS using the Everyday Argument Assessment Task, which has been used extensively with human participants. LLMs first rated the veracity of 10 political claims that were either left- or right-leaning and then evaluated the quality of arguments supporting them. Most LLMs exhibited moderate left-leaning biases with regard to how they evaluated the veracity of claims and the quality of arguments. However, we did not find evidence that LLMs' veracity ratings of a claim predicted how arguments in support of that claim were rated. These findings suggest some popular LLMs exhibit a left-leaning bias towards socio-political discourse.