Proceedings of the Annual Meeting of the Cognitive Science Society

Understanding algorithms is a central learning goal for those in computational fields and is frequently used as a cognitive benchmark for contemporary large language models (LLMs). We present a hierarchy that characterizes algorithmic understanding in terms of observable performance on both concrete and abstract tasks to juxtapose human and LLM performance and draw conclusions about the latter's understanding of the subject space. We evaluate the hierarchy in a controlled assessment study comparing 43 university students with eight LLMs across five canonical algorithms. The resulting performance profiles reveal that LLMs often excel at tasks dependent on concept retrieval while failing on tasks requiring computational example construction. These differences mirror distinctions in rote versus meaningful learning and suggest that algorithm understanding in humans and LLMs may rely on different underlying resources.