Educational intervention research addresses multiple barriers that under-sourced communities face in education. I test three methods to improve education program design and evaluation to develop programs that have substantial and lasting impacts on student outcomes. In my dissertation, I review current challenges to educational program evaluation research and present three different extensions to the methodologies researchers use to improve their practice. To address these challenges, I draw upon perspectives and methodological innovations from adjacent fields including causal inference methods from policy research, measures from implementation science, and practical insights from research-practice partnerships. I contribute three separate studies as potential extension to different parts of our process to discovering what works, for whom, and when. Study 1 describes methods to over-come the challenge of conducting long-term follow up evaluations by empirically testing different intervention design features and analytical decisions for forecasting medium-term impacts of early skill-building interventions in mathematics. I empirically test different study designs and analytical approaches to determine which combinations improve the accuracy of forecasting the medium-term impacts of math interventions using nonexperimental data. Study 2 embraces the challenge of identifying what mechanism or lever to intervene on and how much nonexperimental data can inform theory of change to design interventions. In study 2, I use the data from the Baby’s First Years Study a longitudinal randomized controlled trial (RCT) of an unconditional cash-gift given to the mothers of newborn children living in poverty. I compare nonexperimental estimates of the impact of income on child development and maternal well-being–using data from the control group–to the experimental estimates from the RCT. This study helps us understand and document the importance and difficulty of formulating interventions from theory and nonexperimental data. I discuss the implications of this work for attempting to craft evidence-based policy based on corresponding experimental and non-experimental estimates. Study 3 embraces the challenge of program effect variation across different contexts. I conduct semi-structured interviews and a comparative case analysis to capture evidence about what adaptations and modifications are made to math programs when researchers iterate through program design and implementation. I describe when the adaptations are made, who decides on making the adaptation, what the adaptation is, where it is adapted, and how it is adapted using the Framework for Reporting Adaptations and Modifications-Enhanced (FRAME-IS; Miller et al., 2021) expanded framework for reporting adaptations. I develop a set of guiding principles that can inform researchers about what adaptations to anticipated when designing and implementing math programs.

Current challenges in child development science and education policy include the nature of heterogeneity in early childhood education (ECE) programs’ long-term impacts and their related explanatory mechanisms. Building on theories of skill-building and institutional gateways, the present dissertation addresses some of these challenges. More specifically, I investigated impacts’ variations relative to changes in counterfactual conditions, replicating and extending on a previous ECE program’s evaluation. Then, I tested the hypotheses that factors producing general gains yield impacts that persist, whereas factors producing specific gains yield impacts that fade. I did so by examining the properties of an ECE intervention’s impact on a broad range of cognitive skills. Finally, I attempted to explore some of the patterns underlying the “dark matter” mediating the long-run impacts of ECE interventions by analyzing patterns in the patterns of indirect effects of two ECE programs on educational attainment. In the first study, I replicated Deming’s Head Start study (2009), and appended 10 additional years of data. Using a sibling-comparison design, I replicated positive long-term impacts of Head Start observed in a previous analysis but found no statistically significant impacts on earnings and mixed evidence of impacts on other outcomes when estimated later into adulthood. I repeated the analysis with later cohorts, born to older mothers, and find mostly null and sometimes negative impacts. Finally, combining all cohorts showed generally null impacts on school-age and early adulthood outcomes. I found suggestive evidence that the heterogeneity of Head Start impacts across cohorts was due largely to changes in counterfactual conditions of household characteristics within the sample, notably mother’s age at child’s birth. In the second study, I probed the hypothesis that persistence is unlikely when gains are specific to trained skills and distinguishable from impacts on general cognitive ability. I re-analyzed data from the randomized Abecedarian intervention to probe whether its persistent impact on cognitive skills was linked to impact on cognitive subtests shared variance (i.e., on psychometric g factor). Conducting various measurement invariance tests, I found that impacts were broadly distributed and not distinguishable from impacts on psychometric g. Additionally, I estimated that a subtest-specific impact not-well captured by psychometric g faded. These findings supported the raising IQ/raising g distinction hypothesis, which posits that impacts on cognitive skills are more likely to persist if they generalize across psychometric tests than if they can only be detected on a subset of tests. Breadth of impact might be a worthwhile theoretical and practical proposition in the study and design of ECE interventions with persistent impacts. The final study considered mechanisms translating ECE programs’ initial impacts into longer-term effects. Proposed explanatory pathways leave much of the total impact on long-term outcomes unexplained (so-called ECE intervention “dark matter”). Unidentified mediators are more likely to be proximal if closely aligned with intervention’s features, and more likely to be distal if measuring processes like skill-building or institutional gateways. Thus, leveraging panel data, I considered a series of mediators, grouped as a set across various time-points. Inferences were made about unexplained effects of the “dark matter” from patterns of indirect effects via these observed mediators bundles. Indirect effects were relatively stable, accounting for up to 50 and 20 percent of the impact on educational attainment from Abecedarian and Head Start, respectively. By implication, effects from the “dark matter” were also relatively stable. These findings were consistent with the hypotheses that earlier skills contributed to later institutional gateway process; and that pathways, possibly coupling environmental dimensions with behavior, ran parallel to our chosen mediating bundles and ought to be specified. The final chapter concludes with discussing some of the implications raised by the three studies. I also proposes two lines of research aiming to further address ECE-interventions’ efficiency and long-run impact mechanisms through the life-cycle.

Working memory has been broadly implicated in mathematics performance, but some of the underlying mechanisms of how working memory interacts with math learning are not well understood. This dissertation looks across theories of working memory, to evaluate the competing claims regarding its influence on mental arithmetic. I attempt to reconcile differences between correlational and experimental literatures to better understand the specific impacts of working memory on arithmetic by employing meta-analytic and experimental methods following a dual-task paradigm. Dual-task experiment involve the performing a primary task (e.g., solving simple arithmetic problems) while simultaneously performing a secondary working memory task (e.g., recalling a string of letters or numbers). If both tasks use overlapping cognitive resources then primary task performance will get worse as the secondary task becomes more demanding. Some argue that working memory operates within the dual-task paradigm through domain-specific overlap (e.g., visuospatial tasks overlap with subtraction more than multiplication) while others argue it is more domain-general where more demanding tasks consume more cognitive resources from attention. In order to investigate whether working memory is causally linked to mental arithmetic performance, I conducted a meta-analysis on dual-task experiments, summarizing the effects of secondary task load on mental arithmetic performance (Chapter 1). In addition, I tested a number of relevant moderators, including the type of secondary task load, variations in primary arithmetic, difficulty, and a proxy for task combinations using authors’ predictions. While results supported a robust causal effect of working memory, it was unclear if arithmetic performance was affected purely by the cognitive demands of the tasks or if they were also affected by similarly shared resources with the secondary task. Thus, I conducted a registered report in which I probed whether arithmetic operations are differentially impacted by various types of working memory secondary tasks by replicating an influential dual-task experiment and testing other relevant factors that predicted differences in dual-task performance (Chapter 2). I ran a within-subject experiment and tested whether there was differential interference of verbal and visuospatial loads on multiplication and subtraction performance. I investigated whether the differential effect could be observed across a number of subsample analyses (e.g., comparing across sub-populations and difficulty levels). Prior results from the influential experiment were not replicated, so I conducted further analyses including an additional task to test whether different theories of working memory could reliably predict any kind of interaction between working memory and arithmetic tasks (Chapter 3). Then, I tested main effects and interactions between secondary task type, arithmetic operations, and difficulty and explored whether such task features are implicated in arithmetic strategy choice. I found strong evidence for main effects of the aforementioned factors on arithmetic but not interactions between them except in the case of an arithmetic-based secondary task load. Overall, this dissertation found more evidence to support attention-focused, domain-general models of the influence of working memory on arithmetic performance rather than content-based, domain-specific models discussed within many previous dual-task studies. Lastly, I discuss the implications of these results as well as an outlook for future research.