UC Berkeley

Building a machine learning model is an iterative process: train, evaluate, identify failures, and improve. Benchmarks once anchored this loop, but as models have improved and deployment has expanded, evaluation has become the bottleneck. Metrics fail to capture what practitioners care about, failures are hard to diagnose without manual inspection, and preference signals from users are noisy and hard to decompose. Model development is now constrained not by capability, but by our ability to measure and understand model behavior.To understand your model, you must understand your data. By systematically analyzing the data that flows through models (their outputs, the real-world conversations users have with them, and the training examples that shaped them) we can characterize what people actually use models for, identify where and why models fail, and understand what drives their behavior.We develop a suite of methods that operationalize this view across the model development lifecycle. To diagnose output-level differences, we introduce VisDiff, which automatically describes differences between image sets in natural language, and CompCon, which dis- covers input-dependent divergences between generative models. To build better evaluation data, we create Arena-Hard, a pipeline for curating high-quality benchmarks from crowd- sourced interaction data, and VisionArena, a dataset of 230K real-world user–VLM con- versations with preference labels. To connect evaluation signals to model behavior, we intro- duce VibeCheck, which discovers and quantifies the qualitative stylistic axes that predict user preferences. Finally, SAE Data Analysis shows that sparse autoencoder embeddings provide a unified, interpretable, and cost-effective infrastructure for all of the above tasks, enabling practitioners to trace observed behaviors back to the training data that caused them. Together, these contributions reframe the evaluation bottleneck as a solvable data analysis problem.