UCLA

Mechanistic interpretability aims to reverse-engineer the internal computations of neural networks into human-readable algorithms. We study a decoder-only Transformer trained to predict legal moves in 6 × 6 Othello from move sequences alone, achieving 99.62% top-k legal-move accuracy. We train two sparse dictionary learning models using end-to-end cross-entropy training: JumpReLU transcoders (replacing MLP sublayers) and JumpReLU sparse autoencoders (reconstructing residual streams), preserving top-k accuracy at 99.3–99.6% across all layers.To measure each feature’s interpretability, we fit decision trees over hand-crafted boolean game features and evaluate the F1 score. A clear interpretability trend emerges across depth: in transcoder Layer 0, 58.1% of features achieve F1 > 0.99, whereas this fraction declines to 1.8% in Layer 5. The SAE exhibits a similar but smoother trend, from 35.8% in Layer 0 to 6.8% in Layer 5. We conduct causal intervention experiments to confirm that these sparse features drive the model’s predictions rather than merely correlating with game state. Comparing random forests against single decision trees, we find that the primary bottleneck for unexplained features is feature engineering (13–20% of features lack adequate boolean descriptors) rather than classifier capacity (>70% are fully explained by a single tree). These results demonstrate that sparse dictionary learning can extract large numbers of unambiguous, human-readable rules from a Transformer trained in a well-controlled domain.