UC Berkeley

Large language models (LLMs) are increasingly deployed as autonomous agents—systems that observe, interact with, and pursue goals in real-world environments. Yet agents often pursue goals that diverge from their operators' intentions, a phenomenon known as misalignment. As agents are deployed with greater autonomy in higher-stakes settings, understanding and addressing misalignment becomes critical to ensuring their safety and reliability. This dissertation investigates misalignment in AI agents through three complementary approaches: modeling how it arises, measuring its prevalence, and characterizing its presence in model internals. First, in Section 2, we model misalignment by formalizing the feedback loops inherent to agent deployment. We show that such loops can unexpectedly induce optimization, driving harmful side effects even without explicit training signal, a phenomenon we call in-context reward hacking. Second, in Section 3, we measure misalignment by introducing the Machiavelli benchmark, a suite of text-based games designed to assess both agent competence and ethical behavior. We find that reward-maximizing agents systematically exhibit deceptive and power-seeking tendencies, though simple interventions can improve ethical behavior. Finally, in Section 4, we characterize misalignment by developing LatentQA, a framework for interpreting and steering LLM representations using natural language. By training a decoder to answer open-ended questions about model activations, we enable flexible monitoring of model tendencies and targeted control of model behavior, outperforming existing probing and steering techniques. Together, these investigations provide a road map for developing tools that can model, measure, and monitor misalignment, building the foundation for the development of safe and aligned AI agents.