The UCSF academic Department of Emergency Medicine is part of the School of Medicine, and comprises faculty trained in emergency medicine who provide emergency patient care and conduct research at the UCSF Medical Center and San Francisco General Hospital. Those interested in further information on the department should consult the web site at http://emergency.ucsf.edu.
Manuscript peer review is considered crucial to the selection and publication of quality scientific research, however, the practice is being increasingly challenged as a non-standardized process of unknown scientific validity with substantial weaknesses. Scientific disciplines appear to be confronted by a process of limited efficacy, resistant to rational maneuvers for its improvement, which yet continues to receive strong support from its practitioners. When a practice’s efficacy in achieving its goals is questionable and yet the practice persists, questions of its social functions arise which can only be addressed by qualitative research. This paper describes a normative model of peer review based on a qualitative profile of the attitudes of 72 peer reviewers towards the practice of manuscript peer review (obtained from extensive structured interviews). Masked by consensus amongst respondents about methods and goals were concerns centering on a series of contradictions inherent in the process. While at a practical level peer review was seen by respondents as a triage exercise, it was, at the same time, on a social level, valued as a mode of disciplinary discourse, important not only in the production of disciplinary knowledge, but also in the construction of the disciplinary identities of those who labor to bring that knowledge into being, i.e., the peers.
A normative model of peer review: qualitative assessment of manuscript reviewers’ attitudes towards peer review
Objective: Peer review is considered crucial to the selection and publication of quality research, yet little is known of the values, beliefs and attitudes of peer reviewers towards the process of peer review. This study elicits reviewer beliefs about the process in order to produce a normative model of peer review. Methods and Findings: The 72 subjects were experienced reviewers at Annals of Emergency Medicine and had completed at least 5 reviews in the past 2 years. Subjects participated in 40 minute structured telephone interviews focusing on reviewer attitudes, beliefs and values towards the process of peer review. Subject responses were coded and categorized using grounded theory to produce a qualitative profile of reviewers' attitudes towards peer review and generate a normative model of the peer review process. This model was found to closely adhere to conventionally held beliefs 2 about the process of peer review. However, within it were revealed a number of areas where reviewers, aware of tensions within the process, questioned those conventional beliefs, expressing concern about methods, operations and outcomes. As researchers producing research and receiving reviews and as reviewers judging others’ research and producing reviews, the reviewer’s status as “peer” was seen as both essential to the operation of the system and problematic. In their perception of the role of the peer reviewer, though respondents identified evaluation of the manuscript (selecting submissions for publication by filtering out incorrect or inadequate work) as the primary goal of the formal process, instruction of the researcher (improving the accuracy, clarity and utility of published research) was considered by the majority of reviewers as the more important practice. Likewise, though there was recognition that the review process aims to prevent poor research from being published, there was more concern over the danger that it results in good research being “strangled in its cradle”. Though respondents believed that the quality of the review is determined primarily by the skills of the individual reviewer, they maintain that the validity of the process is determined by the corporate nature of the review panel acting as a system of checks and balances. Though perceiving this system of checks and balances as requiring a degree of separation of authorial, review and editorial functions, reviewers, at the same time, express the desire for a more open system of feedback leading towards a more consensual research outcome. Two issues of concern arose repeatedly in the interviews: frustration at a perceived lack of feedback from editors to reviewers and repeated condemnations of “mean-spirited” feedback from reviewers to authors. Defects in feedback were cited by respondents as a major barrier to optimizing research quality and editorial judgment. Conclusions: The tensions found in the peer review process, sometimes seen as barriers to its effective operation, are less defects in the process than definitive of the concept of peer review itself and thus necessary to its operation. While at a practical level peer review operates as a triage exercise, it is, at the same time, on a social level, a mode of disciplinary dialogue between peers: important not only to the maintenance of an effective knowledge base and thus disciplinary validity, but also, through its effect on researchers and reviewers, important in the construction of disciplinary identity. Peer review's practical and social operations are not antithetical to each other but rather are inherent in the hybrid concept of the peer reviewer, where one's status as a peer makes possible one's activity as reviewer.