Center for Effective Global Action

When determining the effectiveness of a new treatment, randomized trials are not always possible or desirable. The stability-controlled quasi-experiment (SCQE) (Hazlett, 2019) is an observational approach that replaces the usual “no-unobserved confounding” assumption with one on the change in non-treatment outcome between successive cohorts, or the “baseline trend.” We extend this method to allow variance estimation and inference, and apply it for the first time by examining whether isoniazid preventive therapy (IPT) reduced tuberculosis (TB) incidence among 26,715 HIV patients in Tanzania. After IPT became available in the clinics we studied, a non-random 25% of patients received it. Within a year, fewer than 1% of patients on IPT developed TB, compared to 16% of the untreated. Regression adjustment using available covari-ates produces an equally large and highly significant estimate of -15 percentage point (pp) [95%CI: -16.6, -13.7]. While those estimates may generate confidence in IPT’s effectiveness, they cannot eliminate confounding. By contrast, SCQE reveals that the average treatment effect on the treated must be small and indistinguishable from zero, if we assume the baseline trend was flat over the study period. Rather, to argue that IPT was beneficial requires claiming that the (non-treatment) incidence rate rose by at least 0.5 pp per year. This is plausible, but far from certain. The SCQE approach has broad applicability and will sometimes lead to definitive claims of effectiveness. In this case, it usefully aids in protecting against over-confidence in claims that IPT was effective.