Twelve step facilitation (TSF) is an emerging, empirically supported treatment, the study of which will be strengthened by rigorous fidelity assessment. This report describes the development, reliability and concurrent validity of the Twelve Step Facilitation Adherence Competence Empathy Scale (TSF ACES), a comprehensive fidelity rating scale for group and individual TSF treatment developed for the National Drug Abuse Treatment Clinical Trials Network study, Stimulant Abuser Groups to Engage in 12-Step. Independent raters used TSF ACES to rate treatment delivery fidelity of 966 (97% of total) TSF group and individual sessions. TSF ACES summary measures assessed therapist treatment adherence, competence, proscribed behaviors, empathy and overall session performance. TSF ACES showed fair to good overall reliability; weighted kappa coefficients for 59 co-rated sessions ranged from .31 to 1.00, with a mean of .69. Reliability ratings for session summary measures were good to excellent (.69-.91). Internal consistency for the instrument was variable (.47-.71). Relationships of the TSF ACES summary measures with each other, as well as relationships of the summary measures with a measure of therapeutic alliance provided support for concurrent and convergent validity. Implications and future directions for the use of TSF ACES in clinical trials and community treatment implementation are discussed.

This study investigated the correspondence among four groups of raters on adherence to STAGE-12, a manualized 12-step facilitation (TSF) group and individual treatment targeting stimulant abuse. The four rater groups included the study therapists, supervisors, study-related ("TSF expert") raters, and non-project related ("external") raters. Results indicated that external raters rated most critically mean adherence - the mean of all the adherence items - and global performance. External raters also demonstrated the highest degree of reliability with the designated expert. Therapists rated their own adherence lower, on average, than did supervisors and TSF expert raters, but therapist ratings also had the poorest reliability. Findings highlight the challenges in developing practical, but effective methods of fidelity monitoring for evidence based practice in clinical settings. Recommendations based on study findings are provided.

Objective

This study examined the relationships between treatment fidelity and treatment outcomes in a community-based trial of a 12-Step Facilitation (TSF) intervention.

Method

In a prior multi-site randomized clinical trial, 234 participants in 10 outpatient drug treatment clinics were assigned to receive the Stimulant Abuser Groups to Engage in 12-Step (STAGE-12) intervention. A secondary analysis reviewed and coded all STAGE-12 sessions for fidelity to the protocol, using the Twelve Step Facilitation Adherence Competence Empathy Scale (TSF ACES). Linear mixed-effects models tested the relationship between three fidelity measures (adherence, competence, empathy) and six treatment outcomes (number of days of drug use and five Addiction Severity Index (ASI) composite scores) measured at 3 months post-baseline.

Results

Adherence, competence and empathy were robustly associated with improved employment status at follow up. Empathy was inversely associated with drug use, as was competence in a non-significant trend (p=.06). Testing individual ASI drug composite score items suggested that greater competence was associated with fewer days of drug use and, at the same time, with an increased sense of being troubled or bothered by drug use.

Conclusions

Greater competence and empathy in the delivery of a TSF intervention were associated with better drug use and employment outcomes, while adherence was associated with employment outcomes only. Higher therapist competence was associated with lower self-report drug use, and also associated with greater self-report concern about drug use. The nature of TSF intervention may promote high levels of concern about drug use even when actual use is low.