UC Santa Cruz

Reproducibility is the cornerstone of the scientific method. Yet, in the computational and data-intensive branches of science, a gap exists between current practices and the ideal of having every new scientific discovery be _easily_ reproducible. At the root of this problem are the dysfunctional forms of communication between the distinct stakeholders of science: researchers, their peers, students, librarians and other consumers of research outcomes working in ad-hoc ways; groups of individuals organized as independent silos, sharing minimal information between them, all of them with the common task of publishing, obtaining, re-executing and validating experimentation pipelines associated to scientific claims contained in scholarly articles and technical reports. This dissertation characterizes the practical challenges associated to the research lifecycle (creation, dissemination, validation, curation and re-use of scientific explorations) and draws analogies with similar problems experimented by software engineering communities in the early 2000's. DevOps, the state-of-the-art software delivery methodology followed by companies and open source communities, appeared in late 2000's and addresses these analogous issues.

By framing the problem of research delivery (iterating the research lifecycle) as a problem of software delivery, it becomes possible to repurpose the DevOps methodology to address the practical challenges faced by experimenters across the domains of computational and data-intensive science. This thesis presents Popper, an experimentation protocol for writing articles and carrying out scientific explorations following DevOps principles. Popper brings agility to research delivery in a domain-agnostic way by considering the end-to-end research delivery cycle, making it easier for all the stakeholders of science to publish, access, re-execute and validate experimentation pipelines. This dissertation also presents reusable tools in the domain of computer systems research. In a research delivery context, this toolset covers the multiple phases of the research lifecycle and helps systems research practitioners carry out validations of scientific claims in this domain in an agile manner.

For this work, I was awarded the 2018 Better Scientific Software Fellowship, an initiative from the US Department of Energy whose goal is to foster best software development practices among the US scientific community. The Popper CLI tool, and its associated education materials, are currently being used to train new generations of scientists on how to be aware and guard against the multiple practical challenges in reproducibility.