An Analysis of the Top-cited Articles in Emergency Medicine Education Literature

Introduction Dissemination of educational research is critical to improving medical education, promotion of faculty and ultimately patient care. The objective of this study was to identify the top 25 cited education articles in the emergency medicine (EM) literature and the top 25 cited EM education articles in all journals, as well as report on the characteristics of the articles. Methods Two searches were conducted in the Web of Science in June 2016 using a list of education-related search terms. We searched 19 EM journals for education articles as well as all other literature for EM education-related articles. Articles identified were reviewed for citation count, article type, journal, authors, and publication year. Results With regards to EM journals, the greatest number of articles were classified as articles/reviews, followed by research articles on topics such as deliberate practice (cited 266 times) and cognitive errors (cited 201 times). In contrast in the non-EM journals, research articles were predominant. Both searches found several simulation and ultrasound articles to be included. The most common EM journal was Academic Emergency Medicine (n = 18), and Academic Medicine was the most common non-EM journal (n=5). A reasonable number of articles included external funding sources (6 EM articles and 13 non-EM articles.) Conclusion This study identified the most frequently cited medical education articles in the field of EM education, published in EM journals as well as all other journals indexed in Web of Science. The results identify impactful articles to medical education, providing a resource to educators while identifying trends that may be used to guide EM educational research and publishing efforts.


INTRODUCTION
Dissemination of educational research evidence is critical to improving medical education and ultimately patient care. One reasonable measure of the impact of a publication is the number of citations a particular work receives. This number is indicative of the dissemination of its results and serves as a measure of the work's service as a foundation for supporting further research.

Munzer et al.
Top Cited Articles in EM Education Literature Publications and the number of citations also serve as important criteria on which faculty are promoted at some institutions, though other institutions do not use number of citations as a promotional criterion. The number of citations does not solely reflect the work of an individual or team, but also serves as a metric for evaluating the research performed at a departmental, institutional, or even national level. 1 The number of citations of publications may provide one measure by which to determine the impact of work. Additionally, other factors, such as funding, are often used to assess productivity in the research community.
Examination of citations is rarely performed except for the purpose of putting together a promotion package. Azer performed bibliometric analyses evaluating the top-cited articles in medical education; 2 however, there is currently no literature describing the top-cited education articles within the field of emergency medicine (EM). This type of intentional examination can have a number of benefits. First, when examining which articles have received the most citations, it becomes possible for researchers to more easily familiarize themselves with the landmark articles within a field. Second, it provides researchers with information on which topic areas, journals, and research methods tend to be more highly cited. This indicates not only the quality of the research but also the translational impact of the work. 3 Third, as funding for medical education research is difficult, by evaluating the funding sources of highly cited articles, this type of examination allows researchers to identify sources of potential funding.
The objective of this study was to identify the top 25 cited education articles in the EM literature and report on their characteristics, as well as the top 25 cited EM education articles in all other indexed journals. We sought to provide clinicians, educators, and researchers with resources for identifying the highest-impact literature in emergency medical education and a database of options to explore when looking to publish within the field of medical education.

METHODS
Within two distinct searches in the Web of Science index, we identified the top 25 articles related to education and EM. The EM journal search was limited to 19 EM journals (Appendix 1). EM-related journals eligible for inclusion were English-language journals indexed within PubMed. Exclusion criteria for these journals included non-English language journals, prehospital journals, and journals with a nonphysician focus. We ran the second search within all other indexed journals, excluding the EM journals that were excluded in the first search in Web of Science.
Inclusion criteria for each individual article were the following: it had to be primarily EM-related or include emergency physician subjects and be relevant to EM education; its content was deemed educational; it had to be published in English and it had to have subjects that included physicians or future physicians. Among the exclusion criteria for individual articles were these: the subjects did not include EM residents or physicians or medical students in an emergency department setting; or research subjects were non-hospital based (such as EMS or community-based teaching).
From May 18-June 2, 2016, the authors used keywords and search tools within the Web of Science database to retrieve the top-cited articles in both categories. The aim of this search was to identify not only the most highly cited education articles published in EM journals but also the most highly cited EM-based articles related to education that been published in other literature. The keywords were for the large part those used by Azer. 2 (See Appendix 1 for search strategy) Articles were placed in descending order of number of citations in an Excel spreadsheet. Two of the authors then independently assessed both search categories and applied inclusion and exclusion criteria Inter-rater reliability among assessors for selection of the top-cited EM articles, calculated using Cohen's kappa, was acceptable (0.69). The top 25 cited articles involving EM and education were identified and placed into a final list ("Top Cited Education Articles in EM Literature" and "Top Cited EM-related Education Articles").
Articles were assigned for review and divided evenly between the author group. Two authors independently reviewed the full text for each article and recorded the following information: (1) article name; (2) first author; (3) source journal; (4) year of publication; (5) number of citations; funding source (if applicable); (6) journal impact factor; (7) journal discipline; (8) article type; (9) educational content; (10) subjects; and (11) research method (if applicable). Each author-pair discussed the outcomes of this data collection to create a consensus. If any discrepancies arose, a third author evaluated the article in question and provided a tiebreaker. Finally, findings were discussed in conference with all authors. Table 1 summarizes the 25 top-cited medical education articles in EM journals.   Table 2 summarizes the 25 topcited medical education articles involving EM in all other journals.  Articles are listed in descending order with a rank from 1-25 based upon the number of citations, as found in Web of Science at the time of the search.

RESULTS
With regards to EM journals, the greatest number of articles were classified as articles/reviews. The most frequently cited article was "Deliberate Practice and Acquisition of Expert Performance: A General Overview" by Ericsson, published in Academic Emergency Medicine in 2008 and based on a consensus preconference. 12 It had been cited 266 times. Six articles were research papers and seven were curriculum, four of which included a research methodology. One article was a needs assessment. The topics included simulation; learning theory; ultrasound; assessment; learner retention; and interprofessional education. The top three most-cited articles all exhibited a focus on learning theory. Table 3 summarizes these results. Articles were published most commonly in Academic Emergency Medicine (n = 18; 72%), Annals of Emergency Medicine (n = 4; 16%); and Resuscitation (n = 2; 8%). The majority of articles (n = 19; 76%) listed no funding, while five articles (20%) received external funding alone. One article received both internal and external funding.
With regards to the articles in other journals, the majority were research papers. There were seven curriculum articles and two articles/reviews. The topics included simulation; professionalism; management practice; ultrasound; assessment; cross-cultural care; error in diagnosis; learner retention; specialty choice; and supervision.
Articles in the second literature search were from a wide variety of journals including Academic Medicine (n = 5; 20%); Medical Education (n = 3; 12%); British Medical Journal (n = 2; 8%); Journal of the American Medical Association; and Pediatrics (n = 2; 8%). The most frequently cited article was "Prospective Analysis of a Rapid Trauma Ultrasound Examination Performed by Emergency Physicians" by Ma, published in Journal of Trauma in 1995 and cited 193 times. Eight articles had no funding (32%), 13 (52%) were funded by external awards, one article (4%) was funded solely through internal grants, while three (12%) received both internal and external funding. Table 4 summarizes these results.
We calculated the Pearson correlation coefficient (r) to determine if the age of the article was correlated to the number of citations received. For the articles in the non-EM journals, there was a negative correlation between the year of publication and the number of citations (r = 0.42), meaning that the more recently published articles were cited less often.
For the EM journals, however, this correlation was not seen (r = 0.2). Articles from Table 2 (Non-EM) published in higher impact journals were cited more often (r = 0.46). This was not the case for articles in the EM journal search (r = 0.03).

DISCUSSION
This study identified the top 25 most frequently cited EM education articles in both EM literature as well as the remainder of journals based in the Web of Science index. The findings of this study provide information regarding pertinent trends and topics in EM education, as noted in Table 3 and  Table 4, while providing an accessible location to identify some of the highest-impact literature within this field. Additionally, it allows us to take note of the journals in which EM education is most often recognized and published, serving as an historical perspective for those seeking to publish work.
It is apparent that there are trends both with regards to the overall field of EM education, as well as the journals in which these articles are published. Non-EM journals have, on average, a higher impact factor (up to 35 for JAMA), indicating that they have a higher number of cited articles and therefore are likely distributed to a wider audience. It then makes sense that, when appropriate, authors would seek to submit articles to a wider-reaching journal. For instance, the largest number of highly cited articles in both groups of journals was simulation. This suggests that simulation is a topic that has both specialty-specific and wide-reaching interest. The top two cited articles focused on simulation located in non-EM journals (Shapiro et al. and Barsuk et al.) are both more highly cited than the top ranked simulation    article in EM journals. An author looking to publish an article involving simulation would therefore need to balance the benefits of publishing within the field of EM, with associated peer recognition, against the benefits of publishing in a journal with higher impact. It is interesting to note that the top two cited articles in this study are located in EM journals. This confounds the idea that a wider audience will provide a greater number of citations overall and is possibly related to specialty association and peer recognition. Table 1 and Table 2 provide information regarding the type of article, with the finding that the vast majority of articles in the non-EM journals were found to be research articles, while EM journals tended to have more articles/ reviews that were highly cited. This suggests that there is a preference in publication for research-driven articles in non-EM journals. In contrast, Ericsson's theory-based article was significant in EM, as was Croskerrv's article indicating that publishing a key learning theory paper in EM may also provide a meaningful foundation. 11,12 The most common EM journal in which highly cited medical education articles were published was Academic Emergency Medicine. Medical education articles previously considered for publication in Academic Emergency Medicine will now be directed to submit to the new journal Academic Emergency Medicine Education and Training. It should be noted that because this journal will not be indexed for several years, articles published in this journal would not have been considered for this ranking list.
Within non-EM journals, roughly half were specialty focused (i.e., Pediatrics, Journal of Trauma: Injury, Infection, and Critical Care) and half were general medical journals. Articles included within specialty journals tended to have a focus that was less specific to the specialty of EM. It was of note that some of these articles chose a generalizable topic  such as simulation or depression and used a subject population that included EM residents as well as other specialties.
The articles published within non-EM journals had a larger number of authors who received funding, whether internal or external funding, than articles published in EM journals. It should be noted, however, that the top-cited articles in both the EM journals and the non-EM journals did not have any funding. While it can be helpful to have the support that funding provides, this finding suggests that unfunded work is worthwhile and can still be impactful.
One goal of education research is to disseminate educational practices. 3 Many articles have been widely disseminated despite not being highly cited. For instance, all EM residencies use the "Standardized Letter of Evaluation" (SLOE), as one way of reviewing potential applicants; however, the paper describing its predecessor, the "Standardized Letter of Recommendation (SLOR) and subsequent SLOE papers would not appear on the top 25 cited list in this article. 53,54 This suggests that citation numbers alone do not provide all information regarding the reach of research being performed.
This study provides a repository for some of the most impactful literature in EM medical education. For example, the articles on deliberate practice and cognitive strategies for debiasing are important foundations for EM education. Additionally, some research articles form the basis for further research and educational development. By collecting these articles in one location, it allows others to discover landmark articles within the field of medical education. It also allows others to identify trends in EM education research, note common funding sources, and advance the field of medical education.

LIMITATIONS
Limitations of this study include the fact that articles were

Munzer et al.
Top Cited Articles in EM Education Literature searched in only one database, the Web of Science. It is possible that a search performed in a different database, such as SCOPUS, may have provided additional articles or slightly different search findings. For instance, the EM journals were identified a priori of the search. The Journal of Trauma -Injury Infection and Critical Care was not originally identified in that list but could be considered an EM journal. We chose to leave it in the non-EM list based on the a priori listing of journals. Additionally, we excluded articles and journals if they were not English language. This may have skewed search results to favor a Western viewpoint while neglecting articles that may have had additional global influence. Another limitation was our attempt to define what constitutes education research as related to education and training. This may have added a measure of subjectivity, although our inter-rater kappa was acceptable. One final limitation is that this article did not identify where articles were cited, and the subsequent reach of these articles, as well as self-citations. As related above, the ability to determine impact based upon citation count alone is difficult as there are widely read articles that are not highly cited. Citation counts do provide a foundation; further research could identify what qualities make an article more likely to be disseminated.

CONCLUSION
This study identified the most frequently cited medical education journals in the field of emergency medicine, published in EM journals as well as all other journals indexed in Web of Science. The results identify impactful articles that are collected in one location, providing a resource to others while identifying trends that may be used to guide emergency medicine educational research and publishing efforts.