UC Berkeley

The rapid advancement of Large Language Models (LLMs), driven by scaling laws and substantial computational investments, has revolutionized artificial intelligence, enabling significant breakthroughs across domains such as conversational AI, programming, and complex reasoning. However, traditional benchmarks, being static and narrowly scoped, struggle to comprehensively evaluate LLM capabilities in real-world scenarios. These limitations are further compounded by potential contamination, where overlaps between training and test data inflate performance metrics and obscure true generalization.

This dissertation tackles these challenges by introducing robust and scalable evaluation methodologies. First, we develop methods to detect and mitigate benchmark contamination, ensuring reliable and trustworthy assessments. To align evaluations with real-world applications, we present Chatbot Arena, a live, crowdsourced platform that gathers real-time human feedback via pairwise comparisons of model responses. By analyzing millions of interactions, Chatbot Arena offers dynamic insights into model performance, emerging as an industry-standard platform for evaluating human preference in conversational AI.

Complementing human evaluation, we introduce LLM-as-a-Judge, an automated framework that leverages advanced LLMs to approximate human judgments, significantly reducing manual evaluation costs while achieving strong agreement with human preferences. Furthermore, we propose the BenchBuilder pipeline, an automated system for creating diverse, high-quality benchmarks from crowdsourced prompts, addressing contamination issues and enabling robust, real-world evaluations.

These contributions establish scalable, dynamic, and human-centric evaluation systems, advancing the development of AI systems that are not only powerful but also reliable, and aligned with human values.