Lawrence Berkeley National Laboratory

The Exascale Computing Project software deployment effort developed and advanced DevOps capabilities. One goal was to enable robust continuous integration (CI) workflows that span the protected high-performance computing environments found within many of the U.S. Department of Energy's (DOE's) national laboratories. This article highlights several challenges encountered with enabling automation, such as charging models for CI jobs and meeting individualized security requirements that revolve around strongly associating running code with a human identity. It also describes how the Jacamar CI tool evolved to meet later requirements and became a key aspect of the solutions currently offered. Derived from this experience, we offer a conceptual framework for understanding current and future CI challenges at DOE facilities and offer suggestions for long-term solutions.