Consensus Guidelines for Digital Scholarship in Academic Promotion

Introduction As scholarship moves into the digital sphere, applicant and promotion and tenure (P&T) committee members lack formal guidance on evaluating the impact of digital scholarly work. The P&T process requires the appraisal of individual scholarly impact in comparison to scholars across institutions and disciplines. As dissemination methods evolve in the digital era, we must adapt traditional P&T processes to include emerging forms of digital scholarship. Methods We conducted a blended, expert consensus procedure using a nominal group process to create a consensus document at the Council of Emergency Medicine Residency Directors Academic Assembly on April 1, 2019. Results We discussed consensus guidelines for evaluation and promotion of digital scholarship with the intent to develop specific, evidence-supported recommendations to P&T committees and applicants. These recommendations included the following: demonstrate scholarship criteria; provide external evidence of impact; and include digital peer-review roles. As traditional scholarship continues to evolve within the digital realm, academic medicine should adapt how that scholarship is evaluated. P&T committees in academic medicine are at the epicenter for supporting this changing paradigm in scholarship. Conclusion P&T committees can critically appraise the quality and impact of digital scholarship using specific, validated tools. Applicants for appointment and promotion should highlight and prepare their digital scholarship to specifically address quality, impact, breadth, and relevance. It is our goal to provide specific, timely guidance for both stakeholders to recognize the value of digital scholarship in advancing our field.


INTRODUCTION
The promotion and tenure (P&T) process requires the appraisal of individual scholarly impact in comparison to scholars across institutions and disciplines.Comparative metrics such as the journal impact factor and the h-index are used to quantify and compare the quality of an individual's scholarship and, therefore, his or her academic merit. 1 As knowledge dissemination methods evolve in the digital era, we must adapt traditional P&T processes to include emerging forms of digital scholarship. 2In this paper, we aim to first situate our readers within the literature on the topic of academic scholarship, after which we will describe the process by which we derived and refined our consensus guideline.Finally, we will outline the recommendations for the use of digital scholarship for academic promotion made by this particular guideline group.

The Evolution of Scholarship
Scholarship is persistently dynamic.Analog technologies progressed from tablet and stone to pen and paper; modern digital scholarship is evolving with blogs, podcasts, and digital journals.Still, the standards for evaluation are consistent and focus predominantly on impact and quality of the scholarship. 3n 1990, Ernest Boyer of the Carnegie Foundation originally redefined scholarship for the professoriate as belonging to one of four types. 4A decade later, Charles Glassick followed up this work by describing criteria for evaluating scholarship. 5,6o further develop nuances around the scholarship of teaching and learning, Lee Shulman and Patricia Hutchings further clarified specific criteria for this subtype of scholarship (to differentiate it from high-quality, scholarly, and evidencebased teaching). 7These foundational concepts are summarized in Table 1 below.
Traditionally, peer-review processes of academic journals served as a safeguard to ensure overall quality, with evaluators deferring to experts and peers within a scholar's domain to provide an appraisal for quality and an estimate of impact.Similarly, bibliometrics of journals (eg, journal impact factor) 8 and number of citations are surrogates for scholarly reach and proof of impact. 3Despite well-described limitations, these metrics are quantifiable and defined processes that are easily compared.Thus, they are highly relied upon by P&T committees to compare scholars from disparate disciplines.When scholarship using new media is produced, it is reasonable to scrutinize the methodology, content, impact, and quality of these new forms of scholarship, such as digital scholarship.Our use of the term "digital scholarship" in this paper reflects original content that is disseminated digitally, whether that content is research, teaching materials, enduring resources, commentaries, or other scholarly work.
It is unsurprising that as the world becomes more digital, so do scholarly contributions. 9Online-only journals, pre-print archives, 10 and post-production, peer-review journals (eg, Cureus) are rapidly changing the landscape of peer-reviewed publication. 11,12Similarly, with the advent of peer-reviewed blogs, 13 self-published peer-reviewed books, 14 and educational resource repositories, we see an increased breadth of expression from those engaging in the scholarship of teaching.These varied forms increasingly mirror the rigor required by Glassick's criteria and Shulman's paradigms. 15,16

Quantity vs Quality
Judging these new forms of scholarship is different.In many ways, with advanced web analytics, it is easier to quantify the reach and attention (eg, pageviews, podcast downloads, IP addresses that have accessed the content, and time on page) of these digital assets.(See Table 2 for common analytics available for new digital scholarship.)For example, the PubMed-indexed repository MedEdPortal provides download analytics of the published resources that aid in describing entries as fitting within the scholarship of teaching.
However, since many disciplines both within and outside of medicine have not yet fully embraced digital scholarship as enthusiastically as emergency medicine (EM) and critical care, 17 it is no surprise that P&T committees do not yet have specific or universal standards for presentation or evaluation of digital scholarship. 18Those without digital scholarship experience may grapple with understanding the nuances of determining impact and quality in this new era, and their lack of understanding may even result in general skepticism of novel products.Thus, fields that have already established robust methods for determining the quality of digital scholarship can lead the way.Since digital scholarship has matured in EM, 19,20 it is appropriate for our field to call for the identification of best practices for evaluating digital scholarship and for consensus in the inclusion of such items in promotions decisions.Specific guidelines for P&T are lacking despite robust digital contributions proliferating among academicians.In this work, we provide a guiding framework for the presentation and evaluation of digital scholarship for the applicant for promotion, referees for the candidate, and members of P&T committees.

METHODS
We conducted a blended, expert consensus procedure using a nominal group process to create a consensus document. 21Invited participants met at the Council of Emergency Medicine Residency Directors (CORD) Academic Assembly on April 1, 2019 (Seattle, WA), to discuss recommendations for evaluation and promotion of digital scholarship with the intent to develop specific, evidence-supported recommendations to P&T committees and applicants.We began with a live, brainstorming event.The meeting notes were compiled by a leadership team and formatted into a collaborative working document.All authors continued formulating this document via a collaborative online authorship using Google Docs (Google LLC, Mountain View, CA). 22

Participants
The participants were selected by the leadership of the CORD Social Media and Digital Scholarship Committee (EB, ZR, AH).Participants were selected based on criteria of known interest or scholarship in the area, national and international level contributions to EM digital scholarship, and availability to attend the conference in person or by phone.Supplemental Digital Appendix A lists original invitation list and individual selection rationale.The complete list of attendees of the inconference proceedings is listed in the acknowledgments.

Procedures
As a large group, the consensus conference participants democratically developed the discussion and brainstorming procedures.Based on suggestions from the floor about previous consensus procedures at other similar conferences, 23,24 our group decided to engage in small-group brainstorming discussions aligned with the expertise and interests of the participants, which was then discussed as a large group and vetted by the Selected as ALIEM AIR Cardiovascular, Non-ACS module 2019. 39his post was deemed to be of an acceptable score within the ALiEM AIR Scoring tool, and was granted the designation "AIR Approved" by the adjudicating group of educators.There is a second tier below, known as "honorable mention" for posts of moderate quality that did not meet the threshold for inclusion.
ALiEM, Academic Life in Emergency Medicine; AIR, approved instructional resources; SAEM, Society for Academic Emergency Medicine

Ideation and refinement
The participants self-identified their areas of expertise or interest, and then separated into three groups based on these content areas using an iterative process to formulate specific recommendations.The three discussion groups were tasked with formulating recommendations for the following: 1.The P&T applicant for promotion of one's digital scholarship; 2. P&T committee members for evaluation of quality of digital scholarship; 3. P&T committee members for evaluation of the impact of digital scholarship.
Small groups presented preliminary recommendations to the entire group and made further revisions via iterative discussion.Participants transcribed an outline of the discussion and final recommendations and agreed upon them in a democratic fashion.Participants self-selected areas of the manuscript to prepare based on expertise, interest and group approval.All members developed the manuscript from the outline via collaborative authorship.All participants contributed to the manuscript, and CORD Social Media and Digital Scholarship committee members (AH, MS, ZR, EB) served as final editors of the manuscript.

Recommendations for Presenting Digital Scholarship for Promotion and Tenure Demonstrate Scholarship Criteria
When presenting digital scholarship to a P&T committee, begin by ensuring and demonstrating that it meets the criteria of scholarship as defined by Glassick and expanded upon by Sherbino and colleagues with regard to social media. 25,26The adapted criteria are as follows: 1) create original content; 2) advance the field of health professions education by building on theory, research or best practice; 3) be archived and disseminated, and 4) provide the health professions education community with the ability to comment on and provide feedback in a transparent fashion that informs wider discussion.In addition, consider providing evidence of archival and dissemination, such as Google Scholar indexing or inclusion of a digital object identifier (DOI).

Provide External Evidence of Impact
Ensure that your digital scholarship is reflected consistently throughout your promotions dossier.Dissemination metrics are important to include as measures of impact.For example, some blog editors will provide information about how many times a post has been accessed and the locations of its readers, if requested for P&T purposes.Such metrics of dissemination and impact should be presented in the dossier as evidence of your professional reputation as a scholar in your field.
Additional metrics include pageviews, downloads, and geographic reach.Other programs assessing the reach of scholarship, such as altmetrics, may also be valuable. 27The Social Media Index is a relatively newer technique to assess the impact of websites and could be used as a surrogate for impact, much the same as a journal's impact factor. 28See Table 2.
Other measures of impact could include letters of support and awards.If permitted by your institution, consider obtaining letters of support with regard to your digital scholarship.You may also consider inviting both peer letters and letters from non-collaborators discussing the dissemination metrics and impact of specific pieces of scholarship, or simply your overall impact.There are also a number of digital scholarship-based awards, which may be of value for demonstrating scholarly impact. 29

Include Digital Peer-review Roles
Include editor or peer-reviewer roles for digital scholarly content in your curriculum vitae (CV) in a similar manner as you would for traditional print literature.It is important to highlight these supporting components of digital scholarship and they should be factored into the P&T decisions. 25

Citing Digital Scholarship
Cite scholarly work on your CV using a consistent format, whether that work was published in a hard-copy journal or as digital content.Reorganize the categories of scholarly publications on your CV to include a section for "Digital Scholarship," which is the appropriate subheading for items such as blog posts, podcasts, and videos.See Table 3 below for example subheadings for the scholarly bibliography of your CV.Include only those items that reflect true scholarship and relate to the health professions or sciences.Do not list citations for personal website posts or other digital content that is unrelated to your academic position.Consistently format your scholarship across all subheadings on your CV following the American Medical Association (AMA) Manual of Style, 10th Ed. 30 The AMA Manual describes the methods for citing scholarship in most of the categories listed.Examples of each citation type are provided above, and selected citations are adapted in Table 4. Digital scholarship is best formatted using the AMA Manual instructions for "Internet Documents." 33Academic Life in Emergency Medicine (ALiEM) also offers guidelines for citing digital scholarship, with examples. 31igital scholarship is often criticized for lack of peer review, which leads to confusion about the quality and integrity of articles published in exclusively online journals.Peer review is a requirement for all journals to be indexed and available on PubMed, including online journals.Research articles published in online-only journals that have a PubMed unique identifier (PMID) 32 should not be listed under "Digital Scholarship," but rather alongside similar scholarly work published in peerreviewed print journals.Regardless of the mode of publication, all peer-reviewed research be listed under the same CV subheading in the "highest" possible category.
Blog posts that are cited under a "Digital Scholarship" CV subheading can be peer reviewed as well.For example, some blogs offer a peer-review process for authors and identify which posts have undergone peer review. 33Therefore, use qualifiers to identify any digital scholarship citation on your CV that was peer-reviewed or invited.These qualifiers may add additional credibility to your scholarship when a P&T committee reviews your CV.

Crafting a Digital Scholarship Mission Statement
A digital scholarship mission statement can provide a framework for your P&T committee to understand and interpret your digital scholarship. 34Akin to the educational philosophy statement of a teaching portfolio, the digital scholarship mission statement provides a lens through which the committee can interpret the congruence and value of your scholarship. 35,36This narrative should articulate the beliefs that drive your digital work in ways that give perspective to your activities and provide consistency with the academic and social media strategies of each institution.Table 5 below lists specific considerations to include.Please see Supplemental Digital Appendix B for a sample narrative.

Use Traditional Frameworks: Harnessing the Teaching Portfolio
We recommend using traditional frameworks to describe digital scholarly activity and support for academic promotion.One such example of this is the teaching portfolio.As not all institutions require a separate educational portfolio, we recommend that you present your digital scholarship  25 Digital scholarship should not replace materials that are typically included in a teaching portfolio, such as course evaluations or other traditional measures of teaching effectiveness.Teaching portfolios should summarize teaching effort and quality that meet the criteria of Boyer's scholarship of teaching. 4,37Within the teaching portfolio, you may reflect and provide exemplars of digital works and curricula that you have created or curated for learners, but you will not actually list item-by-item the digital scholarship you produce; this should take place in the CV.An entry in a portfolio would holistically describe the pedagogical principles behind a digital educational program or innovation (eg, if you are the creator of a popular podcast, you would explain how you developed the podcast, how you engaged stakeholders to develop the podcast, and, if possible, share data to convey its impact at large through analytics).In contrast, entries of digital scholarship on a CV would be entered individually.Table 6 provides some common examples of digital scholarship, and how they might align best with previous descriptions of traditional academic scholarship (as per Boyer, Glassick, Hutchings and Shulman).

Appraising Impact
There are no hard and fast rules for determining impact.Cabrera and his colleagues have previously suggested scalebased assessments of social media-based impact in their 2017 paper. 34They provide ample guidance to promotions committees for comparing size and scale of various media within a specific subtype (eg, international blog vs a local blog).We highly recommend that readers review this article for further guidance.
Another tool is the Social Media Index, which seeks to create an "impact factor"-like metric based on social media followership.This tool would be best used to judge the impact of an entire digital media collection, such as an entire website or podcast.This tool is available online (https://www.aliem.com/social-media-index/) and has been revised and validated against quality metrics within emergency medicine Free Open Access Medicine resources. 38

Appraising Quality
Due to lower barriers of entry allowing digital scholarship to be more easily produced, general skepticism due to less serious, nonmedical online content, as well as pseudoscientific and/or predatory online content, groups have sought to scaffold and support end-users and educators in seeking high-quality online resources. 39,40The online medical education community has worked to quell skepticism by establishing methods to appraise the quality of digital scholarship. 3See below for a list of critical appraisal tools for rating online secondary resources.For those who have been asked to review files as external referees, these tools may be very useful in guiding us toward high-quality educational content from an educator's CV or portfolio.Some scholars in this space have proposed that we move beyond bibliometrics and surrogates for quality (eg, impact factor, citations, altmetrics), and that P&T committees consider applying direct quality assessments to items of interest (eg, applying the revised METRIQ 41 or ALiEM Approved Instructional Resources (AIR series) scores 39 to a few choice works of digital scholarship from a faculty member's CV, or applying the PRISMA 42 reporting guidelines to a few systematic reviews).Equitably applying both descriptive bibliometrics (eg, citation rate, h-index, etc.) and quality audits to all works of scholarship (digital or otherwise) would go a long way to augment P&T processes.Table 2 contains suggested critical appraisal tools to facilitate secondary resource evaluation.

LIMITATIONS
The live conference was limited to invited participants who could join in person or by phone.Those with scheduling conflicts were therefore excluded from the live session, perhaps limiting valuable insights and contributions.However, those that could not attend the live conference were still heavily involved in the organization and creation of the recommendations postconference via a collaborative writing process.Additionally, all authors participated robustly in the asynchronous editing of this manuscript, reducing the potential that important viewpoints were excluded.Conference participants were selected by the committee members, and important contributors may have been overlooked.To reduce this possibility, invited members were requested to suggest additional invitees.Finally, as digital scholarship participants and creators, there may be bias toward legitimizing our own work over less-familiar scholarship.We attempted to ground our recommendations using best available evidence in order to reduce this bias.However, there is certainly a paucity of literature on how social media is viewed upon (or accepted) as a form of scholarship by the academy.Thus, further explorations of the acceptability or evaluation of digital by P&T committees may be a useful program of research going forward.A paper has recently been published about perceptions in the librarian sciences world that is quite interesting, and worthy of replication within academic medicine. 43

CONCLUSION
As traditional scholarship continues to evolve within the digital realm, academic medicine must also adapt how that scholarship is evaluated.P&T committees in academia are at the epicenter for supporting the changing paradigm in scholarship.Unlike traditional academic products, where reach and impact were difficult to quantify, web-based metrics allow us to track unique users and their locations.The authors suggest that committees critically appraise digital scholarship using the methods outlined in this paper.Applicants for appointment and promotion should highlight and prepare their digital scholarship in a way that specifically addresses quality, impact, breadth, and relevance.It is our goal to provide specific, timely guidance for both stakeholders to recognize the value of digital scholarship in advancing our field.
Audiovisuals, Teaching Materials 8 Media Appearances and Quotes -Print, Television, Online

Table 1 .
Foundations of education and teaching scholarship.

Table 2 .
Summary of metrics used to demonstrate digital scholarship impact, role and quality, with a sample scholarly work.

Table 3 .
Subheadings for "Scholarly Bibliography" section of curriculum vitae.Western Journal of Emergency Medicine

Table 4 .
Suggested examples of digital scholarship citations and qualifier use.Volume 21, no.4: July 2020 Consensus Guidelines for Digital Scholarship in Academic Promotion Husain et al.

Table 5 .
Specific elements to consider within a mission statement.Western Journal of Emergency Medicine