From Guidelines to Interactive Guidance: AI Systems to Scaffold the Academic Peer Review Ecosystem
- Sun, Lu
- Advisor(s): Dow, Steven P
Abstract
Peer review lies at the core of scientific communities, serving as the primary mechanism for evaluating contributions and maintaining research quality. Yet the exponential growth in submissions has pushed many venues to rely heavily on novice reviewers, who often struggle with the complex, context-dependent work of interpreting papers, synthesizing perspectives, and articulating constructive feedback. Existing supports—static guidelines, rubrics, and exemplars—offer limited help for navigating the dynamic, multifaceted nature of scientific authoring and reviewing. This dissertation investigates how intelligent, interactive scaffolding can help novices align more closely with expert practices by embedding dynamic, context-aware guidance directly into their workflows. Drawing on advances in Large Language Models (LLMs), I designed and evaluated three systems that scaffold different dimensions of scientific writing and reviewing. ReviewFlow scaffolds review writing through contextual cues, in-situ knowledge support, and a notes-to-outline synthesis pipeline; a within-subjects study showed that it improved review coverage, quality, and novice confidence. MetaWriter uses extractive and generative summarization to scaffold meta-review writing, helping reviewers reconcile diverse and conflicting opinions. A mixed-methods evaluation found that MetaWriter enhanced clarity and comprehensiveness but also surfaced risks related to bias, over-reliance, and diminished judgment. Extending beyond reviewing, ScienceJury simulates a jury of expert personas to provide authors with diverse, multi-threaded feedback. By offering multiple perspectives and reducing the friction of applying feedback, ScienceJury supported deeper engagement and more strategic revision than single-agent LLM feedback. Together, these systems demonstrate how intelligent scaffolding, grounded in users’ real-time context within the workflow and augmented by LLMs, can support scientific writing and reviewing by enhancing transparency, efficiency, and reflection. At the same time, the studies reveal important risks—including over-reliance, homogenization, and potential erosion of epistemic agency—that demand careful design. This dissertation contributes empirical insights, design principles, and computational methods for building human–AI systems that not only improve scientific reviewing and writing but also promote more reflective, inclusive, and collaborative forms of knowledge production.