UC Berkeley

Inference serving deserves recognition as its own area of systems research. Just as operating systems abstract machine resources, file systems and databases abstract storage— inference frameworks abstract accelerated computing devices for production machine learning workloads. The dissertation makes three primary contributions spanning six years of open-source research in inference systems.First, model composition through Clipper, InferLine, and Ray Serve established principles of SLO-aware pipeline serving—treating serving pipelines as compositional, schedulable units. InferLine achieved up to 7.6× cost reduction through optimized SLO decomposition, and Ray Serve translated these insights into production infrastructure adopted by companies and financial institutions running significant GPU clusters.Second, vLLM—the dominant open-source LLM serving system—introduced Paged-Attention, a virtual memory abstraction for KV caches that achieves near-zero memory waste (<4%) and 2–4× throughput improvements over prior systems. Under the author’s role as one of the project leads (2023–2026), vLLM grew to 75K+ GitHub stars and 2,000+ contributors. The author’s subsequent work extended these ideas through explorations in expanding the scope of PagedAttention to hybrid architectures and NVLink-aware memory management.Third, GPU kernel multiplexing through the OoO JIT compiler and GreenContext pushed resource sharing below the model boundary toward fine-grained hardware utilization via kernel combining and SM partitioning.Together, these contributions demonstrate that purpose-built abstractions are necessary and beneficial at every layer of the inference serving stack—from model composition to memory management to hardware resource multiplexing.