The coronavirus (COVID-19) pandemic resulted in unprecedented challenges for children due to marked emotional distress, social isolation, and substantial grief and loss. Unfortunately, the rise in mental health challenges for youth due to the pandemic could exacerbate the already high rate of unmet treatment needs of youth (Whitney & Peterson, 2019). The pandemic also transformed the field of mental healthcare due to the rapid expansion of telehealth services to slow the spread of COVID-19. Telehealth or telemental health (TMH) services are defined as services that occur remotely through videoconferencing platforms or telephone calls. Although pre-pandemic research on telehealth has been largely positive, the influence of telehealth on youth treatment engagement is unclear due to inconsistent study findings, use of unidimensional measures of engagement, and low utilization of telehealth services in schools (Brooks et al., 2013; Georgeson et al., 2020; Love et al., 2019). Thus, the COVID-19 pandemic offered an excellent opportunity to examine the influence of telehealth on multiple dimensions of engagement within a traditional care setting for youth.

The objective of this two-study dissertation was to investigate the influence of the pandemic and rapid expansion of telehealth on youth treatment engagement using a multidimensional measurement framework. These studies occurred within large urban school-based mental health program serving youth and families with well documented logistical, cognitive, and systemic barriers to engaging in mental health services. The first study examined whether telehealth circumstances facilitated or degraded engagement when measured using a multidimensional and multi-perspective lens. We compared two demographically matched cohorts of families receiving services prior to the pandemic with those enrolled during the phase of the pandemic when stay-at-home orders were in effect, and schooling and non-essential work occurred remotely (hereafter referred to as the mid-pandemic lockdown period). Results revealed that pre- and mid-pandemic ratings of engagement were similar with the exception that caregivers showed greater risk for low attendance and youth had lower expectations about the benefits of mental health treatment within the mid-pandemic lockdown context. We additionally found that the type of delivery mode (i.e., telephone services, videoconferencing services, or a combination of both) was associated with some aspects of engagement for youth, whereas factors unrelated to telehealth were associated with caregiver ratings of engagement during the mid-pandemic lockdown period. The overall results of Chapter 1 suggest that utilization of telehealth services may represent an option for families that does not appear to negatively impact their engagement in services.

Chapter 2 was exploratory and used a mixed methods approach to investigate school-based mental health providers’ early impressions of youth treatment engagement for families enrolled in mental health services during the mid-pandemic lockdown period. A coding system was applied to describe providers’ impressions of low and high engagement indicators based on a multidimensional framework. Multilevel logistic regression models were conducted to examine providers’ abilities to detect engagement challenges and pandemic factors that may influence provider reports of engagement. The qualitative results revealed that providers’ impressions of engagement were substantially more positive than what is described in the literature. The most frequent indicators of engagement reported by providers were associated either with overt behaviors or with ambiguous descriptions that were difficult to code. Use of a multidimensional measurement framework revealed that providers generally appeared to under-detect engagement challenges and use behavioral observation to make engagement inferences. These findings underscore that the telehealth context does not appear to facilitate or degrade providers’ detection abilities, which are generally low, possibly due to a lack of shared vocabulary of engagement indicators, suboptimal assessment strategies regardless of the service context, and/or an over-reliance on the delivery mode as an indicator of engagement in the mid-pandemic lockdown context. Overall, the results of Chapter 2 suggest that providers will benefit from receiving additional supports to enhance early identification of engagement problems and reduce the alarmingly high rates of premature termination.  

Community mental health providers frequently express that emergent life events (ELEs) - acute stressors reported in psychotherapy that negatively impact the client and his/her family - interfere with their ability to implement evidence-based psychosocial treatments (EBTs). Examples of ELEs include death in the family, eviction, and suspension from school. Preliminary data suggest that in a diverse community sample, providers report at least one ELE over the course of treatment for the majority of clients. Furthermore, in sessions with an ELE, providers have significant difficulty carrying out their planned EBT activities.

Building upon these preliminary findings, the goal of this dissertation is to provide an in-depth investigation of ELEs in order to assess the extent to which they disrupt community implementation of EBTs. In Study 1, an observational coding system for ELEs is applied to a large sample of EBT recordings to provide a detailed description of their occurrence and subsequent provider behavior in session. Results demonstrate that ELEs are prevalent and unpredictable, and they often negatively impact providers’ adherence to EBT content. Study 2 uses an observational coding system to investigate how ELEs and other potential barriers to EBT implementation (i.e., barriers to engagement and caregiver participation) are expressed by caregivers in treatment sessions for a collaborative, caregiver-mediated intervention. Findings indicate that ELEs, barriers, and participation may capture a common construct of caregivers speaking up in therapy. Hispanic caregivers were less likely to speak up in therapy, which may have reduced providers’ adherence to the EBT. Finally, Study 3 uses provider- and client-report data to examine associations between ELEs and longer-term outcomes (i.e., provider adherence in later sessions and clinical progress). Results reveal that ELEs negatively impact provider adherence to planned EBT for at least two sessions. Each ELE for which no EBT content is covered significantly reduces youths’ rate of clinical improvement. The collective findings from these studies provide a wealth of information on the complex nature and impact of ELEs. They suggest that in order to optimize the benefits of existing EBTs, structured guidance is needed to assist providers in effectively responding to ELEs.

Well-designed studies have produced over 300 effective treatments that have been summarized in numerous lists of evidence-based treatments (EBTs). At the same time, the field is making great gains in the understanding of how to implement those treatments, once chosen. However, there is no structured guidance for how to select an optimal set of EBTs from those lists that is maximally relevant and minimally redundant with respect to its fit for a targeted service sample. This dissertation introduces relevance mapping, a methodology that addresses this problem. This dissertation consists of three studies that respectively describe the methodology, and use it to evaluate two open questions regarding treatment coordination. Relevance mapping uses automated comparison of the characteristics of each child in a targeted service sample to the participant characteristics from every study of every successful treatment. Relevance mapping addresses who is and is not coverable by any EBT in the literature, under configurable assumptions about which features must match between study participants and children in the service sample. Relevance mapping can then identify the minimum set of treatments needed to serve the maximum number of children in the service sample, based on those same user-defined matching features. The first study describes this methodology in detail along with the context of the problem it addresses within the framework of knowledge management in mental health. The second study compares the efficiency of relevance mapping results when treatments are defined as intact programs or as collections of their constituent procedures. Finally, the third study applies relevance mapping to a large mental health service agency sample to assess the degree to which EBTs fit the problems, demographics, and treatment settings of youths served using wraparound process. Wraparound is a widely implemented and highly popular model for organizing individualized treatments and supports for children with complex needs. However its effectiveness has long been in question, making the combination of wraparound and EBTs and intriguing possibility. The dissertation's overarching goal is to illustrate a methodology for better application of the evidence base to applied settings, under a variety of different definitions and assumptions.

The use of measurement feedback systems, such as clinical dashboards, has been found to improve clinical judgments and client outcomes. However, despite the evidence demonstrating the benefits of tracking and using measurements to enhance treatment, the practice remains relatively rare amongst clinicians, even after they have received training on dashboard use. These challenges highlight the need to investigate the research-practice gap around dashboard utility in order to identify both areas for improvement as well as strengths to harness further. This dissertation sought to explore clinicians’ experiences with clinical dashboards through two studies. The first study examined how various dashboard components affect clinicians’ attitudes towards dashboard use and their abilities to interpret dashboard data effectively. Results found that the presence of advanced dashboard components did not subjectively improve clinicians’ experiences with dashboards. However, expert users were more likely to report that data-rich dashboards were better suited for making clinical decisions. This finding points to potential benefits in varying dashboard complexity around user levels of expertise. The second study used a mixed methods approach to explore barriers and benefits to dashboard use. Qualitative data gathered from supervisor interviews was compared with quantitative data collected from a clinician survey to examine agreements and differences related to continued dashboard use. Feedback indicated broad agreement around certain challenges, such as time constraints and lack of agency support, along with discrepancies around others, with supervisors underestimating the impact on clinicians of low agency-level prioritization. Taken together, the studies comprising this dissertation suggest that dashboard implementation efforts may be improved by designing dashboards flexibly to include content that clinicians find most useful and by targeting agency-level barriers that impede ongoing use in practice.

Well-designed studies have produced over 300 effective treatments that have been summarized in numerous lists of evidence-based treatments (EBTs). At the same time, the field is making great gains in the understanding of how to implement those treatments, once chosen. However, there is no structured guidance for how to select an optimal set of EBTs from those lists that is maximally relevant and minimally redundant with respect to its fit for a targeted service sample. This dissertation introduces relevance mapping, a methodology that addresses this problem. This dissertation consists of three studies that respectively describe the methodology, and use it to evaluate two open questions regarding treatment coordination. Relevance mapping uses automated comparison of the characteristics of each child in a targeted service sample to the participant characteristics from every study of every successful treatment. Relevance mapping addresses who is and is not coverable by any EBT in the literature, under configurable assumptions about which features must match between study participants and children in the service sample. Relevance mapping can then identify the minimum set of treatments needed to serve the maximum number of children in the service sample, based on those same user-defined matching features. The first study describes this methodology in detail along with the context of the problem it addresses within the framework of knowledge management in mental health. The second study compares the efficiency of relevance mapping results when treatments are defined as intact programs or as collections of their constituent procedures. Finally, the third study applies relevance mapping to a large mental health service agency sample to assess the degree to which EBTs fit the problems, demographics, and treatment settings of youths served using wraparound process. Wraparound is a widely implemented and highly popular model for organizing individualized treatments and supports for children with complex needs. However its effectiveness has long been in question, making the combination of wraparound and EBTs and intriguing possibility. The dissertation's overarching goal is to illustrate a methodology for better application of the evidence base to applied settings, under a variety of different definitions and assumptions.

Recent research highlights a growing demand for youth mental health services (Barican et al., 2022; Kazdin, 2019; USPSTF, 2022), prompting the need to enhance mental health workforce capacity. Improving workforce capacity entails strengthening critical decision-making activities, including considering client problems, prioritizing them, and selecting the most suitable practices to address them. Clinical supervision, involving dyads of qualified mental health professionals ("supervisors") and direct service providers ("supervisees"), aims to improve these activities (Proctor, 1986; Milne, 2007). Challenges include time constraints, varying competency activity levels, and difficulty in incorporating new scientific findings, compounded by high turnover rates (Bernstein et al., 2015; Brabson et al., 2020; Chorpita et al., 2021; Collatz & Wetterling, 2012; Dorsey et al., 2017; Powell & York, 1992; Simon & Greenberger, 1971). Integrating decision support systems into clinical supervision could address these challenges, promoting use of evidence and ensuring sustained skill retention among supervisory dyads (Bjork & Bjork, 2020). Within the context of a decision-support system integrated within clinical supervision, this dissertation investigated the reliability of quality, effort, and efficiency metrics, and then examined the associations between ordinal repetition of activities and passage of time with those quality and effort metrics. As such, it explored whether time or repetition is associated with improvement, deterioration, or no change in these metrics. The study analyzes existing data from a multi-site randomized implementation trial aimed at promoting the use of evidence-based methods for engaging youth and families in treatment. We audio recorded and transcribed supervision events in which mental health workers discussed cases at-risk for poor treatment engagement. For part one, 26 supervisees and 17 supervisors discussed 30 cases; for part two, 48 supervisees and 16 supervisors, trained and using a decision-support system, discussed 118 cases. Observational coders rated efficiency and the extensiveness of decision-making activities using a subset of the ACE-BOCS coding system (Chorpita et al., 2018). Efficiency was rated holistically for each event on a 5-point scale, from presence of extensive discussions on unnecessary topics (1) to swift and organized decision-making and planning (5). Quality was evaluated using a dichotomous scale, based on whether each activity met sufficient quality criteria, primarily indicating the presence of the activity. Effort was measured by the total number of words spoken for each activity. Two overall effort scores were calculated based on the total words spoken and duration of the entire event. The total number of supervisory events per supervisory dyad was an indicator of repetition of supervisory activities, and the total weeks since training in the decision-support system measured the passage of time. To assess interrater reliability across all coders, we used Fleiss' kappa (κ) for the four dichotomous quality metrics and ICCs (model [2,1], consistency) for the ordinal efficiency metric. To examine possible change in outcomes, we used mixed effects regression models, examining three hierarchical levels: cases nested within supervisees nested within supervisors. Thus, supervisors were the main level of analysis. We assessed the impact of each level on results and simplified the model if it didn't improve it. To manage skewed data with quality and effort measures having excess zeros, we implemented corrections like the Firth logistic regression and employed specialized models such as the Hurdle model, respectively. These strategies helped mitigate bias and stabilize parameter estimates. Interrater reliability estimates showed that coders consistently rated both the decision-making activities and overall efficiency reliably. A strong positive correlation confirmed the initial validity of the effort measure. Findings revealed changes in efficiency, the presence of quality, and the likelihood of putting in effort as dyads moved through each level of supervision for their cases (for example, from the first supervision event type to the second and then to the third type). Increasing repetition of supervision events or time within each supervision stage did not predict whether the dyads improved in these outcomes. This study underscores the sustainability of quality, effort, and efficiency across repeated supervision events within different supervision types and over time. It also identifies areas for further investigation, including the need for more nuanced and robust measures of quality and effort. Future research should address these issues and explore alternative assessment methods to gain a deeper understanding of workforce learning. This understanding will inform strategies aimed at maximizing workforce capacity to meet the growing demand for high-quality youth mental health services.

Although there is a substantial evidence base of interventions to treat child and adolescent mental health conditions, these interventions remain underused in community mental health settings. Increasingly, clinical supervision has gained attention as an opportunity for promoting use of interventions supported by research. At the interpersonal level, supervisors are well-positioned to support ongoing skill development of therapists through such strategies as role plays, skill modeling, and performance feedback. Supervisors are also well-positioned to impact therapists’ experience of the organizations in which they work. This is particularly important given the high stress, burnout, and organizational turnover characteristic of community mental health agencies. Specifically, supervisors are “middle managers,” interfacing both with upper-level management and frontline workers. This position affords the opportunities to facilitate communication from the top down and bottom up.Acknowledging the potential influence of supervisors on the organizations in which they work and on the individual therapists they oversee, this dissertation examined supervisor impact on therapist experience of their work environment (i.e., organizational climate) and on therapist use of practices supported by research. The first study aimed to examine the role of clinical supervisors as drivers of therapist perceptions of organizational climate within supervisory teams. An understanding of the potential impact of the supervisor on therapist perceptions of organizational climate could inform additional interventions to positively influence climate and facilitate improvement in the quality of care delivered. The second study examined supervision strategies that create a sense of accountability informed by a model from the organizational management literature in which accountability is fostered by behaviors that strengthen links between prescriptions (directives to perform a behavior), events (contexts in which the behavior can be performed) and identity (personal qualities and skills). Informed by this model, the second study aimed to identify the presence of supervision accountability strategies and investigate the association between accountability strategies and therapist follow through use of a therapeutic practice identified during supervision in the subsequent treatment session. As a first step in capitalizing on supervision to improve client outcomes, it is important to determine how supervision influences therapist behavior during treatment by directly testing supervision strategies.

The past 25 years have been characterized by remarkable advances in the development and testing of evidence-based treatments (EBTs) for a variety of youth mental health problems. However, the public health impact of these EBTs has been less than desired or expected. The majority of youth with mental health concerns do not receive EBTs, and when EBTs are delivered in public sectors of care, they have been shown to produce effect sizes well below those seen in randomized clinical trials. A common view among providers serving diverse youth in the community is that EBTs often do not fit their clients’ characteristics, values, and needs. Accordingly, numerous questions have been raised about how to provide mental health services that are both effective and responsive to youth in the community. This dissertation sought to elucidate strategies for improving the utility of EBTs for the diverse youth referred for community mental health services. The first study examined the applicability and implementation of EBTs for youth accessing community mental health services following a county-wide EBT reform initiative. Results showed that a set of more than 30 EBTs had limited applicability to the service sample and that most youth did not receive EBTs delivered with fidelity. In light of these findings suggesting that EBTs may need to be adapted to better fit the needs of the diverse youth in the community, the second study explored providers’ perceptions of barriers and facilitators to engaging traditionally underserved youth in community mental health services. Providers identified barriers to effectively engaging ethnic minority youth as well as families receiving social services. Assigning clients to providers with similar backgrounds, striving for a respectful and nonjudgmental therapeutic style, and making use of implementation supports such as supervision were commonly nominated as strategies for improving client engagement in community mental health services. To identify commonly researched strategies for incorporating culture into psychotherapy, the third study reviewed the literature on psychosocial interventions for ethnic minority youth. This review identified various strategies for incorporating culture into psychotherapy, such as matching clients with providers with similar backgrounds, using handouts that depict ethnic minority youth and families, and allocating time in sessions to discuss issues of prejudice and discrimination. These strategies for culturally tailoring treatment were featured in one-third of effective psychosocial interventions, although most of these interventions targeted disruptive behavior problems among Black and Latinx youth. Findings from these three dissertation studies indicate that the diverse youth seen in community mental health settings may benefit from the selective adaptation of EBTs and highlight opportunities for enhancing the effectiveness and responsiveness of mental health care for traditionally underserved youth and families.

Given the established underutilization of evidence-based practices (EBPs) for youth mental health disorders, the effective dissemination and implementation of EBPs has become a focal point for researchers, policymakers, and administrators. Los Angeles County (LAC) is on the leading edge of policy-influenced mental health service delivery reform. While a surge in EBP training and implementation support offers community providers new opportunities to incorporate EBP into their clinical work, these providers also find themselves left to navigate an increasingly complex and demanding practice environment. During this period of significant change in the nation’s largest county mental health system, it is critical to understand the experiences of the providers tasked with delivering innovative services to those in need. This dissertation sought to explore providers’ implementation experiences through a series of three studies. The first study examined differences among provider attitudes toward six specific EBPs being implemented in LAC, demonstrating that providers could reliably differentiate the appeal and limitations of the various EBPs over and above their attitudes toward EBP in general. Additionally, appealing features of an EBP were associated with self-reported use of that EBP. Study two used an inductive coding approach to explore providers’ open-ended feedback on their training and implementation experiences in LAC, revealing a predominance of negative comments overall. However, the valence of feedback varied considerably across response categories, with unanimous negativity regarding the local treatment context as compared with more balanced comments about the fit and therapeutic consequences of available EBPs. Study three employed a focus group approach to solicit provider experiences implementing an innovative modular EBP during a randomized clinical effectiveness trial in Los Angeles. Qualitative feedback indicated that the treatment’s fit to diverse client populations and ongoing consultation support were of central importance during the implementation process. Design-focused and supportive strategies to address provider concerns were proposed. Taken together, the studies comprising this dissertation suggest that stakeholders may improve the implementation process through enhancing EBP design in accordance with provider feedback (e.g., improving client fit), addressing contextual demands that interfere with providers’ ability to implement EBP, and continuing to invite provider feedback throughout the implementation process.

Despite a wealth of effective psychotherapies for youth mental health disorders, a significant proportion of affected youth do not receive treatment (Merikengas et al., 2011). Although a variety of factors may contribute to this treatment gap, one proposed hypothesis is a lack of caregiver mental health literacy. Mental health literacy (MHL) is defined as the “knowledge and attitudes about mental disorders that aid their recognition and treatment” (Jorm, 2000). Mental health literacy is a relatively new construct, and therefore, there are few psychometrically validated measurement tools available. Furthermore, no validated measures have been developed specifically to assess knowledge about youth mental health problems, and few studies have focused on measurement of caregiver MHL in Latinx samples. Additionally, although some emerging literature focuses on interventions to improve mental health literacy for specific populations, there are no interventions to date that specifically meet the needs of caregivers in Latinx communities, where negative attitudes about mental health problems and treatment is high but service utilization is low (Alegr�a, Vallas, & Pumariega, 2010; Turner et al., 2015). Using a mixed-methods approach, the goals of this dissertation were to: 1) develop an objective measure of MHL that specifically assesses youth mental health problems; 2) validate this measure in a caregiver sample, with a particular focus on Latinx caregivers; 3) examine the relationship between MHL and related constructs, including stigma toward having a child with mental illness and attitudes about seeking treatment for a child with mental health problems in this sample; and 4) explore preferences for and acceptability of interventions to improve caregiver MHL in Latinx communities. Study 1 described the initial development and validation for a newly developed mental health literacy test (MHLT v1). The MHLT v1 consisted of 40 statements that were presented in a True/False/Unsure format and assessed knowledge specifically about youth mental health problems. This measure was piloted in an undergraduate sample (N=354) and results demonstrated that the MHLT v1 had acceptable internal consistency and strong construct validity. Study 2 built upon these findings and discussed the refinement of the MHLT and psychometric validation of this measure in a caregiver sample. One-hundred and sixty-nine caregivers completed the revised MHLT (v2), along with two measures of attitudes about having a child with mental illness and seeking professional help for child’s mental health needs. Demographic variables of interest (e.g., race/ethnicity, education, income), history of mental health diagnoses and service use, and most helpful and trusted information sources, were also collected. Results demonstrated that the MHLT v2 demonstrated good internal consistency. Across demographic variables, non-Latinx White participants outperformed Latinx participants, and participants with graduate education scored higher than participants with a bachelor’s degrees or less. Scores on the MHLT were negatively correlated with stigma about having a child with mental illness, but not significantly correlated with stigma about help-seeking. In Study 3, ten Latinx caregivers participated in a one-time, individual interview and provided feedback about the MHLT content and test-taking experience. Participants reported that the MHLT was manageable and provided suggestions to refine item content. Additionally, interviews explored caregivers’ perceptions about their own knowledge of children’s mental health problems, stigma, and attitudes about help-seeking. Participants in this sample reported low levels of stigma and high openness to seeking professional help for their children’s mental health needs. Finally, participants discussed the perceived helpfulness and feasibility of interventions to improve caregiver mental health literacy, and provided input to improve the acceptability and effectiveness of knowledge interventions for Latinx caregivers in their community. All participants reported that increasing caregiver MHL was important and/or would be helpful or useful. However, participants were mixed about how other community members would respond to programs. Suggestions to increase community engagement and acceptability included delivery by a mental health professional, offering programs in Spanish and English, and reducing use of stigmatizing language (e.g., mental illness) in program content and advertisements.