Background

Systematic reviews are a cornerstone of evidence-based medicine but are useful only if up-to-date. Methods for detecting signals of when a systematic review needs updating have face validity, but no proposed method has had an assessment of predictive validity performed.

Methods

The AHRQ Comparative Effectiveness Review program had produced 13 comparative effectiveness reviews (CERs), a subcategory of systematic reviews, by 2009, 11 of which were assessed in 2009 using a surveillance system to determine the degree to which individual conclusions were out of date and to assign a priority for updating each report. Four CERs were judged to be a high priority for updating, four CERs were judged to be medium priority for updating, and three CERs were judged to be low priority for updating. AHRQ then commissioned full update reviews for 9 of these 11 CERs. Where possible, we matched the original conclusions with their corresponding conclusions in the update reports, and compared the congruence between these pairs with our original predictions about which conclusions in each CER remained valid. We then classified the concordance of each pair as good, fair, or poor. We also made a summary determination of the priority for updating each CER based on the actual changes in conclusions in the updated report, and compared these determinations with the earlier assessments of priority.

Results

The 9 CERs included 149 individual conclusions, 84% with matches in the update reports. Across reports, 83% of matched conclusions had good concordance, and 99% had good or fair concordance. The one instance of poor concordance was partially attributable to the publication of new evidence after the surveillance signal searches had been done. Both CERs originally judged as being low priority for updating had no substantive changes to their conclusions in the actual updated report. The agreement on overall priority for updating between prediction and actual changes to conclusions was Kappa = 0.74.

Conclusions

These results provide some support for the validity of a surveillance system for detecting signals indicating when a systematic review needs updating.

Background

The prevalence of knee osteoarthritis (OA)/degenerative joint disease (DJD) is increasing in the USA. Systematic reviews of treatment efficacy and adverse events (AEs) of hyaluronic acid (HA) injections report conflicting evidence about the balance of benefits and harms. We review evidence on efficacy and AEs of intraarticular viscosupplementation with HA in older individuals with knee osteoarthritis and account for differences in these conclusions from another systematic review.

Methods

We searched PubMed and eight other databases and gray literature sources from 1990 to December 12, 2014. Double-blind placebo-controlled randomized controlled trials (RCTs) reporting functional outcomes or quality-of-life; RCTs and observational studies on delay/avoidance of arthroplasty; RCTs, case reports, and large cohort studies and case series assessing safety; and systematic reviews reporting on knee pain were considered for inclusion. A standardized, pre-defined protocol was applied by two independent reviewers to screen titles and abstracts, review full text, and extract details on study design, interventions, outcomes, and quality. We compared our results with those of a prior systematic review and found them to be discrepant; our analysis of why this discrepancy occurred is the focus of this manuscript.

Results

Eighteen RCTs reported functional outcomes: pooled analysis of ten placebo-controlled, blinded trials showed a standardized mean difference of -0.23 (95 % confidence interval (CI) -0.45 to -0.01) favoring HA at 6 months. Studies reported few serious adverse events (SAEs) and no significant differences in non-serious adverse events (NSAEs) (relative risk (RR) [95 % CI] 1.03 [0.93-1.15] or SAEs (RR [95 % CI] 1.39 [0.78-2.47]). A recent prior systematic review reported similar functional outcomes, but significant SAE risk. Differences in SAE inclusion and synthesis accounted for the disparate conclusions.

Conclusions

Trials show a small but significant effect of HA on function on which recent systematic reviews agree, but lack of AE synthesis standardization leads to opposite conclusions about the balance of benefits and harms. A limitation of the re-analysis of the prior systematic review is that it required imputation of missing data.

Background

Systematic reviews (SRs) can become outdated as new evidence emerges over time. Organizations that produce SRs need a surveillance method to determine when reviews are likely to require updating. This report describes the development and initial results of a surveillance system to assess SRs produced by the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) Program.

Methods

Twenty-four SRs were assessed using existing methods that incorporate limited literature searches, expert opinion, and quantitative methods for the presence of signals triggering the need for updating. The system was designed to begin surveillance six months after the release of the original review, and then ceforth every six months for any review not classified as being a high priority for updating. The outcome of each round of surveillance was a classification of the SR as being low, medium or high priority for updating.

Results

Twenty-four SRs underwent surveillance at least once, and ten underwent surveillance a second time during the 18 months of the program. Two SRs were classified as high, five as medium, and 17 as low priority for updating. The time lapse between the searches conducted for the original reports and the updated searches (search time lapse - STL) ranged from 11 months to 62 months: The STL for the high priority reports were 29 months and 54 months; those for medium priority reports ranged from 19 to 62 months; and those for low priority reports ranged from 11 to 33 months. Neither the STL nor the number of new relevant articles was perfectly associated with a signal for updating. Challenges of implementing the surveillance system included determining what constituted the actual conclusions of an SR that required assessing; and sometimes poor response rates of experts.

Conclusion

In this system of regular surveillance of 24 systematic reviews on a variety of clinical interventions produced by a leading organization, about 70% of reviews were determined to have a low priority for updating. Evidence suggests that the time period for surveillance is yearly rather than the six months used in this project.