The evidence-based policy movement promotes the use of empirical evidence to inform policy decision-making. While several social science disciplines are undergoing a 'credibility revolution' focused on openness and replication, policy analysis has yet to systematically embrace transparency and reproducibility. We argue that policy analysis should adopt the open research practices increasingly espoused in related disciplines to advance the credibility of evidence-based policy making. We first discuss the importance of evidence-based policy in an era of increasing disagreement about facts, analysis, and expertise. We present a novel framework for 'open' policy analysis (OPA) and how to achieve it, focusing on examples of recent policy analyses that have incorporated open research practices such as transparent reporting, open data, and code sharing. We conclude with recommendations on how key stakeholders in evidence-based policy can make OPA the norm and thus safeguard trust in using empirical evidence to inform important public policy decisions.

Objective

Little evidence exists about effective and scalable methods for meaningful stakeholder engagement in research. We explored patient/caregiver experiences with a high-tech online engagement approach for patient-centered research prioritization, compared their experiences with those of professional stakeholders, and identified factors associated with favorable participant experiences.

Methods

We conducted 8 online modified-Delphi (OMD) panels. Panelists participated in 2 rating rounds with a statistical feedback/online discussion round in between. Panels focused on weight management/obesity, heart failure, and Kawasaki disease. We recruited a convenience sample of adults with any of the 3 conditions (or parents/guardians of Kawasaki disease patients), clinicians, and researchers. Measures included self-reported willingness to use OMD again, the panelists' study participation and online discussion experiences, the system's perceived ease of use, and active engagement metrics.

Results

Out of 349 panelists, 292 (84%) completed the study. Of those, 46% were patients, 36% were clinicians, and 19% were researchers. In multivariate models, patients were not significantly more actively engaged (Odds ratio (OR) = 1.69, 95% confidence interval (CI), 0.94-3.05) but had more favorable study participation (β = 0.49; P  ≤ .05) and online discussion (β = 0.18; P  ≤ .05) experiences and were more willing to use OMD again (β = 0.36; P  ≤ .05), compared to professional stakeholders. Positive perceptions of the OMD system's ease of use (β = 0.16; P  ≤ .05) and favorable study participation (β = 0.26; P  ≤ .05) and online discussion (β = 0.57; P  ≤ .05) experiences were also associated with increased willingness to use OMD in the future. Active engagement was not associated with online experience indices or willingness to use OMD again.

Conclusion

Online approaches to engaging large numbers of stakeholders are a promising and efficient adjunct to in-person meetings.

Background

The prevalence of knee osteoarthritis (OA)/degenerative joint disease (DJD) is increasing in the USA. Systematic reviews of treatment efficacy and adverse events (AEs) of hyaluronic acid (HA) injections report conflicting evidence about the balance of benefits and harms. We review evidence on efficacy and AEs of intraarticular viscosupplementation with HA in older individuals with knee osteoarthritis and account for differences in these conclusions from another systematic review.

Methods

We searched PubMed and eight other databases and gray literature sources from 1990 to December 12, 2014. Double-blind placebo-controlled randomized controlled trials (RCTs) reporting functional outcomes or quality-of-life; RCTs and observational studies on delay/avoidance of arthroplasty; RCTs, case reports, and large cohort studies and case series assessing safety; and systematic reviews reporting on knee pain were considered for inclusion. A standardized, pre-defined protocol was applied by two independent reviewers to screen titles and abstracts, review full text, and extract details on study design, interventions, outcomes, and quality. We compared our results with those of a prior systematic review and found them to be discrepant; our analysis of why this discrepancy occurred is the focus of this manuscript.

Results

Eighteen RCTs reported functional outcomes: pooled analysis of ten placebo-controlled, blinded trials showed a standardized mean difference of -0.23 (95 % confidence interval (CI) -0.45 to -0.01) favoring HA at 6 months. Studies reported few serious adverse events (SAEs) and no significant differences in non-serious adverse events (NSAEs) (relative risk (RR) [95 % CI] 1.03 [0.93-1.15] or SAEs (RR [95 % CI] 1.39 [0.78-2.47]). A recent prior systematic review reported similar functional outcomes, but significant SAE risk. Differences in SAE inclusion and synthesis accounted for the disparate conclusions.

Conclusions

Trials show a small but significant effect of HA on function on which recent systematic reviews agree, but lack of AE synthesis standardization leads to opposite conclusions about the balance of benefits and harms. A limitation of the re-analysis of the prior systematic review is that it required imputation of missing data.

While the social sciences have made impressive progress in adopting transparent research practices that facilitate verification, replication, and reuse of materials, the problem of publication bias persists. Bias on the part of peer reviewers and journal editors, as well as the use of outdated research practices by authors, continues to skew literature toward statistically significant effects, many of which may be false positives. To mitigate this bias, we propose a framework to enable authors to report all results efficiently (RARE), with an initial focus on experimental and other prospective empirical social science research that utilizes public study registries. This framework depicts an integrated system that leverages the capacities of existing infrastructure in the form of public registries, institutional review boards, journals, and granting agencies, as well as investigators themselves, to efficiently incentivize full reporting and thereby, improve confidence in social science findings. In addition to increasing access to the results of scientific endeavors, a well-coordinated research ecosystem can prevent scholars from wasting time investigating the same questions in ways that have not worked in the past and reduce wasted funds on the part of granting agencies.

Journal editors have a large amount of power to advance open science in their respective fields by incentivising and mandating open policies and practices at their journals. The Data PASS Journal Editors Discussion Interface (JEDI, an online community for social science journal editors: www.dpjedi.org ) has collated several resources on embedding open science in journal editing ( www.dpjedi.org/resources ). However, it can be overwhelming as an editor new to open science practices to know where to start. For this reason, we created a guide for journal editors on how to get started with open science. The guide outlines steps that editors can take to implement open policies and practices within their journal, and goes through the what, why, how, and worries of each policy and practice. This manuscript introduces and summarizes the guide (full guide: https://doi.org/10.31219/osf.io/hstcx ).