UC Berkeley

There are large gaps in information currently about the way evaluations of multi-level interventions are conceptualized and implemented in public health practice. Existing evaluation literature provides little guidance to evaluators as they attempt to design and implement comprehensive evaluations of such interventions. Multi-level interventions that attempt to create change or impact at two or more levels are being developed and implemented with increasing frequency in several fields, including public health. However, these interventions are often not comprehensively evaluated. If evaluation is conducted, it is often piece-meal, and important elements such as data collection on all levels intervened are often lacking. The reasons behind this disconnect between the comprehensiveness of the intervention and evaluation is not clear.

This dissertation was conducted to fill a current gap in the fields of evaluation and public health. This study expands knowledge by having assessed current multi-level projects and their context, in order to better understand contextual elements described as influencing the comprehensiveness of their evaluations.

The author reviewed literature and current evaluation practice around multi-level evaluations and developed a Framework for Comprehensive Evaluation of Multi-Level Interventions (Framework). A multiple case study was conducted with four projects evaluating multi-level interventions in tobacco control and childhood obesity to further understand contextual elements influencing their evaluation. Analysis of case data took place individually for each project, across projects within the same field, and across the fields of tobacco control and childhood obesity. Case data were later compared with the Framework, to assess how contextual elements influenced evaluations of projects studied.

Cases studied provided information on strengths and challenges of their evaluations, sources of information used in the development of their evaluations, and contextual elements that influenced their evaluations.

Great overlap was found between cases and across the fields studied. These projects developed their evaluations using similar sources of information and expertise. Also, many similar elements were described as influencing the comprehensiveness of their evaluations. While the original Framework and hypotheses developed early in the study were not fully supported, case data provided valuable information on nuance around contextual elements, such as how different elements influenced evaluation development versus evaluation comprehensiveness.

Ultimately, the combination of having strong internal capacity for evaluation, access to external evaluation expertise, adequate and consistent funding for the evaluation, a logic model and other sources of information to guide the evaluation, and flexibility to modify the evaluation as needed, were found as key factors for the success and comprehensiveness of projects studied. This pattern was found across the cases and across both fields.

Field itself may not be a particularly distinguishing factor for cases studied in evaluation development, comprehensiveness, or contextual elements influencing these areas. Projects studied may be providing guidance to others based on their successes around evaluation comprehensiveness, expertise, and amount of time they have been funded and evaluating their programs, rather than their field. And it may be that childhood obesity as a newer field for multi-level interventions, and with large amounts of new funding, is looking to others, especially other programs with many years of multi-level intervention and evaluation experience for more guidance at this time.

Recommendations were developed to address challenges identified, promote promising strategies, and provide considerations for future work. Recommendations were developed to address areas including: capturing synergy, determining "who gets credit", funding issues, access to evaluators and improved training of evaluators, reporting and communication, building an evidence base, use of logic models, inclusion of external evaluation experts, building internal capacity, and flexibility to modify evaluations as needed.

Information gathered through this dissertation can be used by multiple audiences to improve evaluations of multi-level interventions in the future. Multi-level projects, such as those studied, are not carried out quickly or simply, so having multiple suggestions for improvement, to help with the development, comprehensiveness and other areas of the evaluation, can be quite helpful. The ultimate goal of this dissertation was to provide the fields of evaluation and public health with information to increase comprehensiveness and quality of evaluations for multi-level interventions in the future.

Future work beyond this dissertation can go in several directions, including advancing recommendations provided, further study of elements within the Framework, or studying contextual elements beyond those in the Framework, which may also influence evaluation comprehensiveness. Additionally, future studies could look at other cases within these fields, or fields beyond tobacco control and childhood obesity, that are carrying out multi-level interventions, to see if similar patterns are found.