UC San Diego

Large language models (LLMs) have achieved remarkable progress on a wide range of tasks by learning from vast collections of human-written text. This training paradigm instills in LLMs the ability to produce fluent, contextually appropriate language—but it also imposes a fundamental constraint: the model learns to imitate the outcomes of human reasoning, as recorded in text, rather than the cognitive processes of reasoning itself. As a result, LLMs face persistent difficulties with tasks that require structured exploration, grounded symbolic computation, or deliberate planning—capabilities that, in humans, operate partly outside of language.This dissertation explores three distinct approaches to move language model reasoning beyond the boundaries imposed by language imitation. First, we reframe LLM reasoning as a planning problem. Reasoning via Planning (RAP) repurposes the same language model as both a reasoning agent and a world model, enabling structured exploration of the reasoning space via Monte Carlo Tree Search. By explicitly simulating world states and evaluating rewards, RAP enables LLMs to backtrack from wrong reasoning steps and find high-quality solutions— achieving results that surpass GPT-4 with only a 33B-parameter model on plan generation benchmarks.Second, we address the limitation that language text cannot teach symbolic computation. ToolkenGPT augments frozen LLMs with external cognitive tools by treating each tool as a special “toolken” embedding appended to the model’s vocabulary. This lightweight approach— training only the toolken embeddings while keeping the LLM frozen—allows models to master arbitrarily many tools in a plug-and-play fashion, improving performance on numerical reasoning, knowledge-based question answering with over 200 API tools, and embodied plan generation.Third, we question whether language tokens are the right medium for reasoning at all. Chain of Continuous Thought (COCONUT) trains LLMs to reason in a continuous latent space by feeding the last hidden state directly back as the next input embedding, bypassing the language model head. This simple modification gives rise to an emergent breadth-first-search-like reasoning pattern: the continuous thoughts can encode multiple candidate reasoning paths simultaneously, allowing the model to delay commitment and converge on correct answers with fewer tokens than language-based chain-of-thought.Together, these three works demonstrate a systematic progression away from language imitation as the sole basis for machine reasoning: from structured search over language reasoning paths, to grounding in symbolic external tools, to reasoning freed from language tokens entirely.