- Main
Treating Distillation as Pedagogy: Simplicity and Scaffolding in Large Language Models
Abstract
Human pedagogy relies on scaffolding to tailor information complexity to a learner's capacity, yet current AI knowledge distillation often subjects resource-constrained student models to computational cognitive overload via unstructured, verbose reasoning. To investigate the alignment between human learning principles and machine learning dynamics, we propose a unified computational framework that treats distillation as a controlled cognitive experiment. We manipulated the granularity and sequence of the pedagogical signal across two distinct tasks. Our results demonstrate two key phenomena paralleling human cognition: a simplicity effect, where concise elucidations significantly outperform complex expert rationales by minimizing extraneous cognitive load, and a scaffolding effect, where a simple-to-complex curriculum proves essential for convergence in reasoning tasks. These findings provide computational evidence that less is more in the context of knowledge distillation for resource-constrained models, suggesting that effective alignment in student-teacher paradigms requires a shift from maximizing information quantity to optimizing pedagogical quality.