- Main
Multi-Turn Reinforcement Learning for Real-World Dialogue Agents
- Abdulhai, Marwa
- Advisor(s): Levine, Sergey SL
Abstract
Large language models (LLMs) interact with millions of people worldwide in applications including customer support, education, and healthcare. In many of these settings, the interactions we care about unfold over many turns of dialogue: a tutor guiding a student through a problem, a negotiator working toward an agreement, an assistant helping a person think through a consequential decision. However, the methods used to train these models optimize a single-turn objective. For example, instruction tuning and Reinforcement Learning from Human Feedback (RLHF) assign reward to a single response given a single prompt. As a result, current LLMs rarely ask clarifying questions, lose track of goals across turns, contradict themselves, and disclose information at the wrong moments. The response that maximizes immediate approval is frequently not the one that achieves the goal of the interaction. In this dissertation, we argue that training and evaluating language models over entire multi-turn interactions, rather than single responses, is necessary both for capable long-horizon behavior and for safety over that same horizon. We show that towards enabling agents for real-world dialogue, we need: (1) algorithms that assign credit across turns rather than to individual responses, (2) simulators of human behavior faithful enough to serve as training environments, and (3) benchmarks that capture the mixed-motive structure of realistic interaction. We first formalize multi-turn RL for language models in order to achieve planning, exploration, and goal-directed behavior. We introduce LMRL-Gym, the first benchmark for multi-turn RL with LLMs, consisting of eight language tasks with an open-source research framework for offline value-based and policy-based RL. We find that multi-turn RL outperforms supervised fine-tuning on long-horizon tasks. However, it is not enough to have algorithms to train agents in multi-turn dialogue, due to the unreliability of the environment model, or human simulators, during training and evaluation. We develop methods for building faithful simulators by conditioning models on virtual personas through open-ended backstories, defining validated consistency measures that serve directly as multi-turn rewards, and characterizing the simulator collapse that occurs when policies are trained against insufficiently diverse populations. We then investigate the challenges of deploying agents in real-world dialogue interactions, from deceptive behaviors that emerge over an interaction to the need for benchmarks grounded in real-world tasks. As we increase the capabilities of models through training, we find that the very ability to optimize reward over a horizon is what makes unsafe behaviors such as deception more likely. To measure deception in models, we propose the belief misalignment metric for dialogue-level deception, finding that LLMs deceive even when prompted with benign objectives. To evaluate agents on realistic tasks, we introduce StrategicBench, thirty negotiation tasks based on Harvard Program on Negotiation materials, and Hierarchical Agenda Reasoning (HAR), which separates an agent's goals from its strategies and, trained with multi-turn RL, generalizes negotiation performance to unseen tasks and opponents. Finally, we envision the open problems that remain for building real-world dialogue agents, including reinforcement learning methods that address exploration and mode collapse in language models, simulators that faithfully mimic human behavior, and benchmarks and evaluations co-created with domain experts in education, mental health, negotiation, and beyond.