Academic Primer Series: Key Papers About Peer Review

Introduction Peer review, a cornerstone of academia, promotes rigor and relevance in scientific publishing. As educators are encouraged to adopt a more scholarly approach to medical education, peer review is becoming increasingly important. Junior educators both receive the reviews of their peers, and are also asked to participate as reviewers themselves. As such, it is imperative for junior clinician educators to be well-versed in the art of peer reviewing their colleagues’ work. In this article, our goal was to identify and summarize key papers that may be helpful for faculty members interested in learning more about the peer-review process and how to improve their reviewing skills. Methods The online discussions of the 2016–17 Academic Life in Emergency Medicine (ALiEM) Faculty Incubator program included a robust discussion about peer review, which highlighted a number of papers on that topic. We sought to augment this list with further suggestions by guest experts and by an open call on Twitter for other important papers. Via this process, we created a list of 24 total papers on the topic of peer review. After gathering these papers, our authorship group engaged in a consensus-building process incorporating Delphi methods to identify the papers that best described peer review, and also highlighted important tips for new reviewers. Results We found and reviewed 24 papers. In our results section, we present our authorship group’s top five most highly rated papers on the topic of peer review. We also summarize these papers with respect to their relevance to junior faculty members and to faculty developers. Conclusion We present five key papers on peer review that can be used for faculty development for novice writers and reviewers. These papers represent a mix of foundational and explanatory papers that may provide some basis from which junior faculty members might build upon as they both undergo the peer-review process and act as reviewers in turn.


INTRODUCTION
Peer review is a key component of academic publishing, and aims to provide rigor and relevance to the publishing process. 1 While the primary aim of the peer-review process is to select and prepare manuscripts for publication, the service of peer review also provides professional development, reward, and opportunities for further scholarship to the reviewer. However, faculty new to peer review may feel intimidated or unprepared to engage in this scholarly activity.
While peer review draws upon skills that many faculty already have, it does require content and process knowledge that is rarely formally taught to novice reviewers. Quality peer review does not necessarily correlate with traditional markers of experience, such as academic rank, research training, or grant funding. 2 While peer review has traditionally been a solitary practice, models are emerging that facilitate a mentored or team-based approach. These approaches allow junior faculty to receive mentorship in the one-to-one mentored model, and engage in a community of practice in a team-based approach. 3,4 However, a foundational understanding of the elements of a quality peer review and the peer-review process can be helpful prior to engaging in peer review.
The Faculty Incubator was created by the Academic Life in Emergency Medicine (ALiEM) team to provide early-career educators with a community of practice where they can discuss and debate topics relevant to the 21st century medical educator. To that end, we created a one-month module focused on peer review. This paper is a narrative review that highlights some important literature that may assist junior educators who are seeking to learn more about the peer-review process.

METHODS
In the seventh month of the ALiEM Faculty Incubator (September 2016), we discussed the topic of peer review. We monitored the proceedings of this group of educators from September 1-30, 2016. Our online discussions involved both junior faculty members and faculty mentors. While discussions occurred, we gathered the titles of papers that were cited, shared and recommended within our online discussion forum and compiled these into a list.
This list was then augmented by the following:  Once the augmented list was completed, we then conducted a three-round voting process, similar to our previously described Delphi-inspired method to build consensus around which papers to feature. 5 This was a modified Delphi method since our selection panel was comprised of both novices (i.e., junior faculty members, participants in the Faculty Incubator) and experts in the field (i.e., experienced clinician educators, all of whom have published >10 peer-reviewed publications, who serve as mentors and facilitators of the ALiEM Faculty Incubator). However, we intentionally used this method so as to involve both junior and experienced clinician educators to ensure we selected papers that would be of use to a spectrum of educators throughout their careers.

RESULTS
Our ALiEM Faculty Incubator discussions in combination with expert recommendations and social media calls yielded a total of 24 articles. Our procedure allowed us to create a rank-order listing of all these papers in order of perceived relevance, from the most to the least relevant. The top five papers were expanded upon below. Our ratings of all 24 papers are listed in the Table, along with their full citations.

DISCUSSION
Our group determined the following papers to be of highest interest and relevance to novice reviewers and faculty developers. The accompanying commentaries are meant to explain the relevance of these papers to junior faculty members, and also highlight considerations for senior faculty members when using these works for faculty development workshops or sessions.

Summary
Lovejoy and colleagues provide an overview of the peer-review process for Annals of Behavioral Medicine, describing and providing examples of high-quality reviews for that journal. Although the focus of the process is specific to the behavioral and social-science focus of this journal, the general principles are largely applicable to most academic journals and fields. Specifically, the authors raise awareness of the need for more formal reviewer guidance and attempt to do so by way of this manuscript.
The authors begin by discussing the roles of the editors and editorial board for the journal and laying out the responsibilities of each. Of special interest to potential reviewers is the process by which action editors select reviewers, highlighting the importance for new reviewers to only identify actual areas of personal expertise. The majority of the article focuses on the actual process of reading a manuscript and drafting the review, including specific considerations pertaining to each of the separate sections of a manuscript. The authors provide a framework for critically appraising manuscripts by explicitly highlighting the roles of the reviewer in order to 1) offer opinions on the strengths and weaknesses of a manuscript, and 2) provide guidance to authors in how to improve scientific process and communication. To conclude, the authors summarize the "do's and don'ts" of the peer-review process in addition to providing an annotated example of a high-quality review from a paper published previously within the journal.

Relevance to Junior Faculty Members
This paper is relevant to junior faculty who wish to participate in peer review for service, personal professional development, and as a scholarly activity for career advancement. The paper provides an understanding of the peer-review process that is crucial to being able to perform the responsibilities of a reviewer. Although not the focus of the article, a common mistake made by novice peer reviewers is overextending themselves: This may include attempting to review beyond the limits of their actual expertise. A better approach is to select fewer areas of expertise in order to build a portfolio of timely, high-quality reviews, expanding knowledge with progression of one's career, leading to future review opportunities.
The largest area of relevance for junior faculty in this article is found in the step-by-step approach to performing a review. The authors provide a guide for reviewers, starting with accepting or declining an invitation to review and concluding with review submission. Additionally, reviewers should consider reading the articles at least once without marking or making comments, just to assess for readability and understanding.
This article then provides a concise yet complete series of considerations for each section of a manuscript, which can help guide the novice reviewer's thought process and ultimately drafting of the review. Additionally, the article provides two different options for organizing the review, highlighting the necessity to identify major versus minor concerns. Novice reviewers may find this article a useful guide, providing a framework for initial reviews that will likely become more intuitive with experience and time.

Considerations for Faculty Developers
This paper provides a useful "how-to" resource for faculty developers to prepare academic faculty for peer review. It is a broad and comprehensive overview that provides both step-by-step instructions on the process, and examples to highlight how to apply these instructions to an actual review.

Summary
As part of the Twelve Tips series, this paper provides valuable advice for the more novice peer reviewer. The authors discuss the importance of gauging your qualifications, any significant biases, and available time prior to agreeing to review (Tips 1, 3, 5, and 6). They also emphasize the role of the reviewer, not only in critically appraising the article itself, but also in determining how well the submission fits within the journal's style and mission (Tips 2 and 4). They address the importance of confidentiality and professionalism, highlighting the need to keep critiques constructive and reminding the reviewer that the purpose is to strengthen the paper (Tips 7, 8, and 9). The last three tips are, perhaps, the most valuable of all. Tip 10 addresses confidential comments to the editor, clarifying what should be included and the importance of consistency between these recommendations and those shared with the authors. Tip 11 emphasizes the differences between educational and basic scientific research, reminding those reviewing in education journals the differences in approaches and limitations. Finally, tip 12 provides a variety of strategies to improve one's peer-review skills. While isolated interventions have not significantly influenced peer-review skills, 7-10 using this combination of strategies may be more fruitful.

Relevance to Junior Faculty Members
As junior faculty members become involved in peer review, it is important to keep some core components in mind. This paper provides a concise table highlighting key questions for each component of the submission. The table in this paper can serve as a simple one-page guide for the more novice reviewer to help structure his/her reviews. The subsequent tips emphasize some of the less tangible, but equally important, components of the review process. From a professionalism standpoint, this paper reminds the potential reviewer that s/he should ensure that s/he is adequately qualified and unbiased, and able to provide constructive criticism, rather than simply highlighting faults. The paper also highlights the importance of providing an overview, general recommendations, and assessment of suitability for the journal in addition to the discussion of specific suggestions. Finally, the paper highlights numerous strategies for improving one's peer-review skills. Examples include attending peer-review workshops, reading papers highlighting strategies for producing high quality reviews, reading other reviewers' comments from the same paper, asking for feedback from the editor and colleagues, and reflection on one's experiences.

Considerations for Faculty Developers
This paper provides a helpful roadmap to guide and orient novice reviewers to the many steps and factors impacting the peer-review process, and many of these are concisely summarized in the Table. Tips 2 ("Familiarize yourself with the journal style") and 11 ("Know the differences between educational and scientific research") highlight the value of mentorship for novice reviewers, as these subtle differences in article types may not be immediately apparent to those who are less familiar with the medical education literature, and reviewers may feel ill-prepared to critique research approaches that fall outside of their more traditional biomedical training. Guiding novice reviewers to be introspective about both potential conflicts of interest (Tip 3) and bias (Tip 6) are essential mindsets, and allowing time for reflection (Tip 5) will encourage reviewers to provide the most thoughtful and nuanced suggestions for improvement.

Summary
This study sought to explore the strengths and weaknesses of submissions after analyzing peer-reviewer ratings and comments. A content analysis of the 151 peer-reviewed research manuscripts submitted to the 1997 and 1998 Association of American Medical College-sponsored Research in Medical Education (RIME) conference was performed. Peer reviewers for RIME come from around the world, and all accepted manuscripts are published in a supplement of Academic Medicine. Each masked submission was evaluated by four or five reviewers who work as medical educators. Anonymous comments and a review form are completed by each reviewer. Eight areas are rated on a five-point scale (excellent, good, fair, unsatisfactory and not acceptable). The eight areas rated are problem statement and  background, research design, sampling, instrumentation and data collection, results, conclusion, writing and importance. Finally, each reviewer is asked to use a four-point (definitely include; acceptable, probably include; questionable, probably exclude; definitely exclude) global rating and give additional comments on merits or shortcomings of submission. Interestingly, nearly two fifths of the reviewers recommended rejection without any unsatisfactory ratings on the checklist. The top reason for rejection was inappropriate, incomplete or insufficiently described statistics. This was followed by over-interpretation of results. The top reason for manuscript acceptance was importance, timeliness, relevance, and critical pertinent problem. Both good and bad quality of writing was raised by many reviewers, stressing the importance of wellwritten manuscripts. Acknowledging limitations rather than ignoring them was also deemed important. As summarized by Bordage, "scientific writing demands both good science and writing good manuscripts."

Relevance to Junior Faculty Members
Bordage highlights important items that junior faculty should consider when taking part in the peer-review process. The ability of a reviewer to determine what is a well-written manuscript and what are appropriate statistics for a study seem most important. This article implies that peer reviewers should have a background in statistics in order to be able to interpret analysis and results as appropriate. A junior peer reviewer based on this study should also be able to critically appraise a research project for its well-performed design, timeliness and novel approach. The ability of the study to provide practical, useful implications should also be considered. The junior reviewer must also be able to provide written comments and feedback to the authors in order to provide guidance in what can be improved.

Considerations for Faculty Developers
This paper is a great launching point for a discussion on how to improve both peer review and quality of writing with junior scientists. By being aware of the common "fatal flaws" encountered in the field of medical education, it is possible to then pay more attention to these problems when reviewing papers. Faculty developers may want to use the lists generated by this paper to create some easy-to-use handouts for guiding junior faculty members when critically appraising their own work as well. Discussions around each of the most common grounds for rejection and acceptance can be used to scaffold journal club proceedings or internal peer-review processes of research units.

Summary
This editorial, written by the editor-in-chief of Medical Education, discusses the importance of understanding and incorporating reviewer comments, even when the author does not entirely agree with them. The author highlights the importance of the peer-review process for improving a manuscript, emphasizing the value in both well-written, high-caliber reviews, as well as those in which the reviewer is unclear or incorrect in their interpretation. In the latter case, Eva emphasizes that peer reviewers are reading submissions much more carefully than the standard readership and that any confusion should prompt the author to reevaluate the text and address any ambiguity. He subsequently discusses the importance of peer review and provides several strategies for improvement, which include the provision of a guideline for reviewers, deliberate feedback, and maximizing opportunities to review.

Relevance to Junior Faculty Members
After devoting significant time and effort to a publication, junior faculty may become frustrated after receiving reviewer critiques, especially when the reviewer expresses confusion over what appeared so clear to the author. This paper reminds the junior faculty member that reviewer comments are valuable both by emphasizing what may have been missed, as well as those aspects which may be unclear to readers. It is advisable after receiving reviewer comments to set the manuscript aside for several days and return after the emotions have passed and empathize with the reviewer's comments and perspective. Junior faculty may also benefit by seeking feedback from colleagues and mentors prior to submission. Finally, when serving as a reviewer, junior faculty should review the existing guidelines and seek feedback to ensure that they continually improve their peer-review skills.

Considerations for Faculty Developers
This paper highlights several important concepts for faculty teaching peer-review skills to others. In particular, adopting the maxims of "Did I learn anything?" and "What could the authors have done to convince me of the argument they are trying to convey?" frames peer review as an activity rooted in the goal of providing actionable feedback to authors that will help them to improve their work (as opposed to simply giving summary judgments on the manuscript's overall quality). The Medical Education reviewer guidelines (www. mededuc.com) highlighted in this article also provide a useful rubric for the types of general issues that reviewers should consider when conducting a review.

Relevance to Junior Faculty Members:
Peer review is an essential part of an academic career and most junior faculty flounder a bit with the first reviews. This article emphasizes the "big picture" of peer review. It is important because it so clearly and simply states the appropriate responsibility and behavior of an excellent and thoughtful reviewer. It provides an easy-to-follow outline of issues that a reviewer must address when evaluating a paper. The most important emphasis of this paper for junior faculty is the advice to not only find flaws in a paper, but help find solutions. This is an essential skill in thinking critically, evaluating scientific literature and in ultimately developing an academic career.

Considerations for Faculty Developers
This is a pragmatic article that can be used as a springboard for discussing the role and integration of peerreview responsibilities for those new to the job. Faculty developers will find this a useful guide for reminding faculty members (who may have experienced the slings and arrows of blinded peer review) about how to provide positive and constructive peer reviews. This paper may be a useful prophylaxis against the negative feelings that can emerge between reviewees and reviewers, reminding us to be empathetic, helpful, and kind -rather than unremittingly blunt, mean, or sarcastic.

LIMITATIONS
As with our previous papers, we did not design this study to be an exhaustive, systematic search of the literature. We attempted to seek assistance with finding more papers by using expert consultation and an open social media call via Twitter using hashtags #MedEd & #FOAMed, which yielded some important recommended papers. Considering the depth and breadth of our final list, we feel that by using these adjunctive methods we have overcome the limitations of our unstructured collection of papers.

CONCLUSION
We provide a reading list on the topic of peer review that may be beneficial as a primer for junior clinician educators and as a potential reading list for senior faculty members leading faculty development efforts. We hope this paper may serve as a guide for clinician educators who are looking to further the development of their own peer-review skills.