UC Berkeley

Over the last few years, the massive scaling up of resources poured into the pretraining of large language models (LLMs) has led to rapid advances in LLM capabilities. However, performance on the most challenging reasoning-heavy tasks scales far too slowly as a function of pretraining alone to achieve reasonable performance at a practical scale. A different approach to scaling up computation is therefore needed in order to meaningfully improve performance on these tasks.In this thesis we aim to understand when and if scaling up computation at test-time with LLMs can fulfill this gap, making for an effective alternative to scaling up pretraining compute. We begin by demonstrating the necessity of scaling up pretraining resources in order to broadly improve model capabilities. From here we turn towards test-time scaling as a potential substitute. In particular, we carry out one of the first careful empirical studies comparing the efficacy of different strategies for scaling up computation at test-time with scaling up pretraining resources. We show that indeed scaling test-time compute with LLMs can be preferable to pretraining in certain settings. However, there remain several important limitations to scaling up test-time compute that make it not quite one-to-one comparable to pretraining. In particular, test-time scaling comes with an increased latency cost and treats each prompt independently, resulting in redundant computation across related queries. Finally, we propose two novel methods - Context Distillation and Sleep-time compute which can be used to overcome these limitations with test-time computation. These techniques work by applying computation ahead of time to prepare for a specific distribution of upcoming tasks, such that at test-time the model can respond quickly without meaningfully sacrificing performance.