An Emergency Medicine Virtual Clerkship: Made for COVID, Here to Stay

Introduction Safety concerns surrounding the coronavirus 2019 pandemic led to the prohibition of student rotations outside their home institutions. This resulted in emergency medicine (EM)-bound students having less specialty experience and exposure to outside programs and practice environments, and fewer opportunities to gain additional Standardized Letters of Evaluation, a cornerstone of the EM residency application. We filled this void by implementing a virtual clerkship. Methods We created a two-week virtual, fourth-year visiting clerkship focused on advanced medical knowledge topics, social determinants of health, professional development, and professional identity formation. Students completed asynchronous assignments and participated in small group-facilitated didactic sessions. We evaluated the virtual clerkship with pre- and post-medical knowledge tests and evaluative surveys. Results We hosted 26 senior medical students over two administrations of the same two-week virtual clerkship. Students had a statistically significant improvement on the medical knowledge post-tests compared to pre-tests (71.7% [21.5/30] to 76.3% [22.9/30]). Students reported being exposed to social determinants of health concepts they had not previously been exposed to. Students appreciated the interactive nature of the sessions; networking with other students, residents, and faculty; introduction to novel content regarding social determinants of health; and exposure to future career opportunities. Screen time, technological issues, and mismatch between volume of content and time allotted were identified as potential challenges and areas for improvement. Conclusion We demonstrate that a virtual EM visiting clerkship is feasible to implement, supports knowledge acquisition, and is perceived as valuable by participants. The benefits seen and challenges faced in the development and implementation of our clerkship can serve to inform future virtual clerkships, which we feel is a complement to traditional visiting clerkships even though in-person clerkships have been re-established.


Emergency Medicine Virtual Clerkship
Villa et al.
Medical students who are EM bound typically complete at least two EM clerkships to gain additional experience and explore varied training environments. Visiting clerkships also provide Standardized Letters of Evaluation (SLOE)crucial to the residency application -and allow for reciprocal exposure between the student and program. [6][7][8] In response to COVID-19 restrictions, many medical schools and residencies pivoted to online education and cancelled visiting rotations. 9,10 With the loss of in-person visiting clerkships, a novel virtual curriculum was urgently needed to fill this void.
As an alternative to traditional in-person visiting clerkships, we created, implemented, and evaluated a virtual "visiting" clerkship with a focus on advanced and less commonly taught topics (ie, social EM and professional development). Based on the most recent recommendations, many institutions have removed restrictions on in-person rotations but continue to limit visiting rotations to one per student. 11 Looking forward, the virtual environment creates a unique opportunity for programs to continue to meet their applicants more in depth, in addition to circumventing geographic and socioeconomic barriers often faced by students participating in traditional visiting rotations.

CURRICULUM DESIGN
Rather than replicating a traditional clerkship virtually, we designed our curriculum to focus on advanced medical topics: social determinants of health; structural competency; and professional identity formation 12,13 by employing Kern's method for curriculum development. 14 We identified educational needs as institutional COVID-19 restrictions were released. We performed a needs assessment including data from our postgraduate year one class as near peers. Our topic list was further refined by consensus among the author group, which included a clerkship director, associate program directors, medical education fellow, and senior EM residents. We developed goals and objectives informed by the topic list and the additional goals of exposing students to our residency program and social EM, as well as advancing professional identity formation. Our traditional in-person sub-internship experience typically covers medical knowledge topics commonly seen in the emergency department as well as skills to help learners thrive while rotating in person. In addition to being vastly different from a traditional experience given that it would be delivered virtually, we felt that this rotation could possibly serve as an ideal environment to cover social EM and professional identity formation, topics that would benefit from minimal interruptions or competing pressures.

OBJECTIVES
Curriculum goals included teaching advanced EM clinical knowledge, introducing social EM and professional identity formation, and providing exposure to our residency program. See Table 1 for course goals and objectives.
When choosing educational methods, we used the conceptual framework developed by Brown et al to maximize online learning and engagement. This framework encourages expectation management, learner engagement, and "nudging." 15 Our orientation outlined expectations, including asynchronous assignments and recommended norms for small group. We prioritized interactive teaching modalities and active learning to maximize engagement such as smallgroup learning among as well as our "Virtual Escape Room" (Appendix A) and simulation. Our small group, case-based discussions used a flipped classroom model, an effective and recommended modality for virtual instruction. [16][17][18][19][20][21][22] All small-group facilitators were reminded of the best practices for online, small group teaching, 23 which included use of introductions, learner-directed questioning to encourage equal participation, and "nudging" -reminders for learners to actively participate (See Appendix B).
Each virtual clerkship session was held on weekdays over two weeks for a total of 10 instruction days. The students were expected to complete various asynchronous learning assignments (estimated two hours daily) and attend four hours of Zoom (Zoom Video Communications, Inc, San Jose, CA) sessions daily. (See Appendix C for example schedule and specific content.) Cases from "Foundations of EM," a national free, open-access online resource, were used to teach medical knowledge. 24 Social medicine instruction was done using modules from the International and Domestic Health Equity and Leadership (IDHEAL) Section from the University of California, Los Angeles. 25 Chosen modules included Language, Incarceration, Gender Identity, Race, and Homelessness, and assigned readings from those modules were delivered to learners via email. The Virtual Escape Room consisted of a tricyclic antidepressant overdose case, created by authors AV and TJ (Appendix A) with inspiration from another published escape room. 26 The virtual simulation was carried out over Zoom, and cases from our traditional clerkship were used, which cover pediatric anaphylaxis, motorcycle trauma, hypothermia, and abdominal aortic aneurysm. The "Communities of Practice Panel" consisted of a panel of faculty/attendings who practice in different EM environments (ie, tertiary referral center, county, community, Veterans Administration, and critical access community). Many of these modalities have previously been highlighted as effective teaching modalities. 27,28 Within the professional identity formation theme, learners read Carol Dweck's "Mindset" to prepare for a book club-type discussion -a modality previously well-received by other learners. 29 Finally, students were introduced to basic pedagogical techniques and practiced non-medical teaching sessions for their peers and faculty; feedback was provided.
Asynchronous assignments consisted of Emergency Medicine Reviews and Perspective C3 podcasts, free openaccess medicine (FOAM) curated by the Academic Life in EM (ALiEM) AIR series, Foundations of EM "Frameworks," and articles introducing topics of social EM. 24,25,30,31 Asynchronous content was designed to correspond to daily synchronous Villa et al.
Emergency Medicine Virtual Clerkship content; specifics can be found on Appendix C. We made our virtual clerkship available to all fourth-year medical students applying into EM via the Visiting Student Application Service (VSAS) website and offered it twice during the 2020-2021 academic year. 33 Given our predicted teaching resources, we estimated an ideal class size of less than 25 students per session. Ultimately, 26 students enrolled (nine in the first session, 17 in the second session). Attendance at all sessions was mandatory.
We recruited a group of residents and faculty to teach for a total of 24 lecturers and 26 small-group facilitators across both sessions. Facilitating the clerkship during the two-week session required one of three clerkship directors to be present on Zoom four hours per day, in addition to administrative tasks related to that day's activities.
We assessed medical knowledge with a 30-item, peerreviewed, multiple-choice test consisting of questions donated by RoshReview (Rosh Review LLC, Huntington Woods, MI), a commercial question bank company. RoshReview validates questions against real-world exam performance such as the in-training exam for EM residents. 32 Questions were chosen by the course directors by a systems-based approach (ie, neurology, cardiovascular, etc) with the goal of choosing questions that were reflective of clerkship's curriculum. Students did not have the ability to see the answers to the questions after taking the pre-test. Because the test was conducted at home, students theoretically could access open-access content in real time. Students completed the same medical knowledge test on the first and last days of the clerkship. 34 We calculated mean scores and compared preand post-tests using a paired t-test, analyzed with the software statistical package Stata 16.1 (StataCorp, College Station, TX). Test scores had no bearing on final grades.
Each two-week clerkship contained five social EM sessions, which were assessed by an anonymous survey exploring previous experience with the topic and comfort with applying content learned to the clinical environment (Appendix D). This tool had been previously used in residency education by the IDHEAL group. It was developed by content experts after literature review to maximize content validity. We analyzed questions relevant to this clerkship. Participants were sent an online anonymous survey after each of the five sessions. In total, we sent a total of 130 surveys (five surveys per participant) and analyzed the data descriptively.
We assessed students' overall attitude toward the course with a 16-item evaluative survey consisting of 11 multiple-choice, one slider scale, and four free-response items. The survey was created by SV, who has a Master's in Education, had advanced training in survey design and experience in qualitative research, and was the course director, all providing content validity  evidence. We read survey items aloud among the author group and piloted the survey with a small reference population to optimize response process validity. The survey was distributed on the last day of class (Appendix E). We calculated and reported descriptive statistics for survey questions with discrete answer choices. For free-response data within the survey, two authors (SV and AV) performed a thematic analysis. SV trained AV, a senior resident, to perform a thematic qualitative analysis. The analysts independently reviewed the data and later met to establish a final coding scheme, which they then independently applied to all data. After applying the final coding scheme, they identified discrepancies and finalized themes. The simple percent agreement between the two analysts was 80.3%. Discrepancies were resolved via in-depth discussion and negotiated consensus. This study was deemed exempt by the University of California, Los Angeles Internal Review Board (IRB #20-002014) approved on November 19, 2020.

IMPACT/ EFFECTIVENESS
Twenty-six students participated in the virtual clerkship representing 22 medical schools and all regions of the US. All students completed the medical knowledge pre-and post-test. Mean test scores improved from 21.5 (standard deviation [SD] +/-2.6) to 22.9 (SD +/-1.24) (P = 0.006), effect size 0.68, 95% confidence interval, 0.12-1.24.
Of the 130 IDHEAL post-module surveys administered, 98 (75%) were completed. Of the modules chosen, incarceration was least likely to have been previously covered with only 6% (1/18) of respondents having prior instruction. Eighty-nine percent (87/98) of respondents "strongly agreed" that these topics were important for patient care in the ED, and 66% (65/98) felt more confident after completing the modules. See Table 2 for full results.
Almost all (25/26, 96%) students completed the end-ofrotation evaluative survey. Of all respondents, 95% "strongly agreed" or "agreed" that the rotation should be repeated in the future, and all "strongly agreed" or "agreed" that the rotation would impact the way they ranked our program. Major themes from the qualitative analysis are described in Table 3.
Prior literature has demonstrated knowledge acquisition and retention from virtual curricula, and we saw similar results, albeit our study demonstrated only a modest improvement. 35,36 One explanation for the lack of larger change is that our assessment items may not have been perfectly aligned with our curriculum as the questions were pulled from a standard question bank. For future versions, we would strongly consider constructing and validating our own internal assessment of medical knowledge to be better aligned with our objectives. Additionally, students rated themselves as more confident in discussing and managing social medicine topics. Ideally, we would be able to conduct a repeat assessment at a predetermined timepoint to assess whether the social EM content had modified their practice as residents. While we emphasized our program's strong social EM vision, other programs may replicate this This topic is important for the care of patients in the ED. 87 (89%) 10 (10%) 0 (0%) 0 (0%) 1 (1%) ED, emergency department.  Instructor orientation Respondent 4: "Whenever you split people into small groups, ALL proctors should implement the round-robin approach for participation AND the proctor should tell the students when their turn is over. Most proctors did this, and I really appreciate it. When it didn't happen, the sessions felt less fluid" Alignment of asynchronous and synchronous content Respondent 9: "Some of the asynchronous resources were not too connected to the sessions that day." Table 3. Themes identified from the end-of-rotation survey.

Emergency Medicine Virtual Clerkship
Villa et al.
curriculum to focus on their own strengths. In the past, visiting clerkships have acted as a recruiting tool for residencies, 8 and virtual clerkships may also allow residencies to highlight strengths and successfully recruit. Students from 22 institutions participated in our clerkship at minimal cost to them (only the cost to apply via VSAS). In contrast, EM applicants averaged 1.9 visiting rotations costing almost $1000 per rotation in 2019. [37][38][39] While many institutions have implemented scholarships for underrepresented in medicine students, virtual clerkships remove financial barriers for all students and may be an invaluable option for students with familial or other obligations. 40 Virtual clerkships represent an additional strategy to help mitigate the socioeconomic barriers of visiting rotations.
While students perceived our virtual experience to be valuable, several challenges were encountered. The clerkship required significant administrative efforts and a large number of facilitators to create the intimate small-group experiences critical to its success. There was no protected time or funding for the instructors. Overall, at least 40 hours per clerkship among NW and SV were required, which did not account for planning and time from all instructors who volunteered their time. These requirements may be adjusted by limiting the number of students enrolled. Furthermore, as we look to the future, simultaneously administrating a virtual clerkship and in-person clerkship will likely require significant additional administrative support.
While this rotation ultimately served 25 students, we considered this rotation to be a success as we were fortunate to match three interns of our current class from the Virtual Clerkship. If we were to repeat this clerkship again, we would expand our evaluation efforts as it was limited, mainly only allowing for assessment in small groups. One other addition would be to encourage asynchronous communication among students and faculty. Examples of communication would be continued improvement of the environment and incorporation of daily questions expanding on the day's content to further enhance spaced repetition. Lastly, interest in our virtual clerkship was likely increased due to COVID-19 restrictions on in-person opportunities. Future versions will require more advertisement and may not bolster as much interest.
We envision we will offer both versions of each clerkship separately moving forward. However, we likely would not offer a formal SLOE to students who pursue the Virtual Clerkship given we cannot comment on their clinical skills in the virtual format. However, we would gladly write a letter of recommendation as, in some ways, program leadership may get to know these students in a more personal way, especially with certain aspects of the SLOE such as "commitment to EM." Finally, we may incorporate some of the social EM content and other teaching modalities into the traditional clerkship.

CONCLUSION
This virtual clerkship was created in response to an acute educational need created by the COVID-19 pandemic.
However, our experience suggests that virtual learning experiences may be valuable in the future as an adjunct to traditional in-person rotations. Virtual rotations provide flexibility allowing for the incorporation of topics not traditionally taught (eg, social EM), allow residencies and students increased access to one another, and may eliminate socioeconomic barriers advancing educational equity.