UC Merced

Large Language Models (LLMs) underpin state-of-the-art progress in natural language processing tasks ranging from creative text generation and summarization to code completion and multi-modal reasoning. Their rapid adoption across industry and academia highlights an urgent need for cost-effective deployment. Unfortunately, the sheer scale of contemporary models—often comprising tens of billions of parameters—translates into memory footprints that can exceed hundreds of gigabytes when stored in FP16, rendering straightforward, single-GPU inference infeasible for many practitioners and cloud tenants. Even well-funded organizations face capacity bottlenecks when attempting to serve long-context workloads or large batch sizes.

Tensor offloading, combined with tensor quantization and asynchronous task execution, provides a feasible and cost-effective solution by utilizing host memory to enable large-scale LLM inference with a limited number of GPUs. However, existing approaches struggle to fully utilize all available computational and memory resources due to a lack of consideration for (1) whether to use quantization and how to apply it effectively, and (2) managing thread-level parallelism within and across tasks. As a result, these approaches provide suboptimal solutions.

This thesis introduces LM-Offload, a performance model-guided framework for LLM inference that integrates quantization-aware tensor offloading with thread-level parallelism control. By using CPU memory to extend limited GPU capacity and managing tensor placement and compute threads effectively, LM-Offload improves both memory efficiency and throughput. The system builds analytical models that estimate the overhead and benefit of quantizing weights and KV cache, and dynamically determines how to best apply quantization strategies during both the prefill and decode phases.

Thread-level parallelism is also carefully optimized. Traditional systems such as FlexGen and ZeRO-Inference fail to utilize CPU threads efficiently, resulting in cache contention and idle resources. LM-Offload introduces an algorithm that models the concurrency of tensor movement and attention computation, ensuring more balanced thread allocation. Profiling results guide this assignment process without introducing runtime latency.

Experimental results demonstrate that LM-Offload outperforms FlexGen and ZeRO-Inference, two state-of-the-art systems for LLM inference, by up to 2.95× (2.34× on average) and 2.88× (1.57× on average) respectively in inference throughput. These results validate the effectiveness of performance-guided quantization and parallelism.

This thesis contributes not only a practical system for efficient LLM inference, but also a methodology for incorporating hardware-aware modeling into inference frameworks. The LM-Offload framework bridges the gap between system-level optimization and LLM-serving challenges, making large-scale models more accessible for deployment across a wider range of hardware configurations.