Department of Emergency Medicine (UCSF)

Objective: Peer review is considered crucial to the selection and publication of quality research, yet little is known of the values, beliefs and attitudes of peer reviewers towards the process of peer review. This study elicits reviewer beliefs about the process in order to produce a normative model of peer review. Methods and Findings: The 72 subjects were experienced reviewers at Annals of Emergency Medicine and had completed at least 5 reviews in the past 2 years. Subjects participated in 40 minute structured telephone interviews focusing on reviewer attitudes, beliefs and values towards the process of peer review. Subject responses were coded and categorized using grounded theory to produce a qualitative profile of reviewers' attitudes towards peer review and generate a normative model of the peer review process. This model was found to closely adhere to conventionally held beliefs 2 about the process of peer review. However, within it were revealed a number of areas where reviewers, aware of tensions within the process, questioned those conventional beliefs, expressing concern about methods, operations and outcomes. As researchers producing research and receiving reviews and as reviewers judging others’ research and producing reviews, the reviewer’s status as “peer” was seen as both essential to the operation of the system and problematic. In their perception of the role of the peer reviewer, though respondents identified evaluation of the manuscript (selecting submissions for publication by filtering out incorrect or inadequate work) as the primary goal of the formal process, instruction of the researcher (improving the accuracy, clarity and utility of published research) was considered by the majority of reviewers as the more important practice. Likewise, though there was recognition that the review process aims to prevent poor research from being published, there was more concern over the danger that it results in good research being “strangled in its cradle”. Though respondents believed that the quality of the review is determined primarily by the skills of the individual reviewer, they maintain that the validity of the process is determined by the corporate nature of the review panel acting as a system of checks and balances. Though perceiving this system of checks and balances as requiring a degree of separation of authorial, review and editorial functions, reviewers, at the same time, express the desire for a more open system of feedback leading towards a more consensual research outcome. Two issues of concern arose repeatedly in the interviews: frustration at a perceived lack of feedback from editors to reviewers and repeated condemnations of “mean-spirited” feedback from reviewers to authors. Defects in feedback were cited by respondents as a major barrier to optimizing research quality and editorial judgment. Conclusions: The tensions found in the peer review process, sometimes seen as barriers to its effective operation, are less defects in the process than definitive of the concept of peer review itself and thus necessary to its operation. While at a practical level peer review operates as a triage exercise, it is, at the same time, on a social level, a mode of disciplinary dialogue between peers: important not only to the maintenance of an effective knowledge base and thus disciplinary validity, but also, through its effect on researchers and reviewers, important in the construction of disciplinary identity. Peer review's practical and social operations are not antithetical to each other but rather are inherent in the hybrid concept of the peer reviewer, where one's status as a peer makes possible one's activity as reviewer.