UC Davis

This dissertation investigates three structural concerns that determine whether modern software engineering can sustain trustworthy computational systems: the reproducibility of software engineering artifacts, the reliability in the large language model (LLM) era of software engineering, and the scalability of modern software infrastructure. Reproducibility determines whether software artifacts and empirical results can remain executable and trustworthy over time. However, while maintaining BugSwarm, one of the software defect datasets that set the foundation for the evaluation of software engineering techniques, the concern of reproducibility of software engineering research became increasingly apparent. To examine this concern, this dissertation presents a comprehensive survey of 151 software defect datasets and the first systematic study of reproducibility across five widely used Java defect datasets. This study finds that reproducibility criteria vary across datasets and that all datasets are prone to breakages. A 13-month longitudinal case study of BugSwarm then shows that 62.6% of artifacts break at least once and identifies the dominant root causes of these breakages. It further demonstrates that dependency caching and artifact isolation can preserve reproducibility at 95%, matching the level achieved by handcrafted curation. To support the continuous growth of reproducible software defect datasets, this dissertation also presents ActionsRemaker, which extends this line of work to GitHub Actions, the dominant CI/CD platform for open-source software. ActionsRemaker demonstrates its ability to reproduce 72.2% of randomly selected fail-pass pairs. It has since been incorporated as a key component of the BugSwarm pipeline, mining thousands of reproducible artifacts to date and establishing continuous integration as a sustainable source of dataset growth. Reliability determines whether software systems, including those that incorporate LLMs, can be analyzed and repaired when they fail. However, due to the probabilistic and black-box nature of LLMs, new challenges arise for the reliability in the LLM era of software engineering. To examine this concern, this dissertation introduces semantic markers, an instrumentation technique that adapts delta debugging to LLM-integrated systems. Semantic markers isolate the input fragments responsible for incorrect outputs without requiring access to model internals. The approach, evaluated on Amazon Web Services production systems, demonstrates that prompt debugging can be transformed from an ad hoc manual process into a systematic procedure. Complementarily, to assess the reliability of LLMs themselves when applied to software engineering tasks, this dissertation also presents CI-Bench, a framework for evaluating LLM-based program repair tools on real CI/CD failures. The evaluation results with CI-Bench reveal that the strongest tool-model pairing repairs only 35% of these failures, quantifying how far LLM-based repair remains from broad real-world CI/CD coverage. Scalability determines whether software infrastructures continue to behave correctly as workloads, data, models, and deployments grow. However, in the shadow of scaling laws, scalability faults emerge as distinct failure modes in modern software infrastructure. To examine this concern, this dissertation conducts the largest empirical study of scalability faults in distributed systems, manually analyzing 444 issues across 10 systems. The study finds that most scalability faults arise from the interaction between dimensional code fragments and a small set of scalability anti-patterns. Building on this finding, this dissertation presents ScaleLens, which combines lightweight dynamic analysis with static pattern matching to detect such faults. ScaleLens identifies 4.2× more dimensional code fragments than a memory-monitoring baseline and surfaces 334 latent issues in the latest Cassandra, HDFS, and Ignite codebases. To extend this characterization to modern AI infrastructure, this dissertation also studies 109 scalability faults across Ray Serve, vLLM, and SGLang along four orthogonal axes. The analysis reveals a control-plane/data-plane bifurcation specific to LLM-serving stacks and identifies a small set of defect categories as the highest-leverage class for systematic detection and repair.