Prospective Randomized Crossover Study of Simulation vs. Didactics for Teaching Medical Students the Assessment and Management of Critically Ill Patients

e Abstract—Background: Simulation (SIM) allows medical students to manage high-risk/low-frequency cases in an environment without patient risk. However, evidence for the efﬁcacy of SIM-based training remains limited. Objective: To compare SIM-based training to traditional didactic lecture (LEC) for teaching medical students to assess and manage critically ill patients with myocardial infarction (MI) and anaphylaxis. Methods: Prospective, randomized, non-blinded crossover study of 28 fourth-year medical students. Students were oriented to the human patient simulator, then randomized to SIM or LEC between August and December 2007. The SIM group learned to manage MI using SIM training and the LEC group learned via PowerPoint lecture. All subjects’ assessment and management skills were then evaluated during a simulation session of MI. During a second instruction session, the students crossed over and were taught anaphylaxis using the opposite modality and similar assessments were conducted. Completion of critical actions for each case were scored, converted to percentages, and analyzed via signed rank test. Results: Of 28 subjects, 27 performed better when trained with SIM compared with LEC ( p < 0.0001). Mean scores were 93% (95% conﬁdence interval [CI] 91–95%) of critical actions completed for SIM and 71% (95% CI 66–76%) for LEC. Absolute increase for simulation was 22% (95% CI 18–26%). For three domains common to MI and anaphylaxis, simulation scores were higher for history (27%, 95% CI 21–38%), physical examination (26%, 95% CI 20–33%), and management (16%, 95% CI 11–21%). Conclusion: SIM training is superior to didactic lecture for teaching fourth-year medical students to assess and manage simulated critically ill MI and anaphylaxis patients. © 2011 Elsevier Inc.


INTRODUCTION
Simulation (SIM) encompasses any technology or process that re-creates a contextual background that allows a learner to experience success, mistakes, receive feedback, and gain confidence in a safe environment (1).This may be particularly useful for low-frequency but highacuity scenarios.SIM tools include, but are not limited to: computers, virtual reality equipment, standardized patients, and high-fidelity mannequins.
The Institute of Medicine, the Educational Technology Section of the 2004 Academic Emergency Medicine Consensus Conference, and the public have advocated for increased SIM-based training in an effort to reduce error (2)(3)(4)(5)(6).Patient SIM has spread from anesthesiology to other disciplines such as internal medicine, pediatrics, neonatology, radiology, trauma, nursing, emergency medical services/disaster medicine, and emergency medicine (EM) (7)(8)(9)(10)(11)(12)(13)(14)(15)(16)(17)(18).Although SIM-based training makes intuitive sense and has shown benefit in other fields such as aviation, the evidence to support its effectiveness in medicine is limited.
Research has not conclusively demonstrated whether SIM-based training is superior to the traditional teaching methods of didactics, or the more recently utilized problem-based learning format (19 -21).We conducted a prospective, randomized, non-blinded crossover study to determine whether SIM-based training is superior to traditional didactic lecture (LEC) training in the assessment and management of simulated critically ill patients presenting with myocardial infarction (MI) and anaphylaxis.

Participants and Setting
We conducted this prospective, randomized, non-blinded crossover study from August to December of 2007 in a simulation laboratory at an urban university teaching hospital.All fourth-year medical students enrolled in a required EM clerkship were eligible without exclusion criteria.The standard didactic core curriculum consisted of PowerPoint-based lectures as well as interactive small group sessions with faculty.Human patient simulation had not yet been part of the core curriculum.Students were randomized by drawing a "number from a hat."Odd numbers received SIM-based training on instruction day 1 and lecture training on instruction day 2, and even numbers received the converse.
During clerkship orientation, students were offered voluntary participation in the study.Use of the simulator was not restricted to the study, and results of the study did not affect clerkship evaluation.The study was approved by the local Institutional Review Board and subjects provided informed consent.

Orientation to the Human Patient Simulator
After randomization, all students received an equivalent 30-min orientation to the human patient simulator (Laerdal SimMan® full scale patient simulator; Laerdal Medical Corporation, Wappingers Falls, NY), which included introducing and reviewing simulator features as well as the physiologic monitoring devices (Figure 1).Students were instructed to verbalize their thoughts, orders, and actions during the simulated patient scenario.By the end of orientation, expectations of the simulation exercise were explained and all medical student questions were answered.

Curriculum Development, Checklists, and Scoring
An advisory committee consisting of six emergency physicians was created to develop and review the teaching ma- Simulation vs. Didactics for Teaching Medical Students terials and assessment tools.Generally, the learning objectives were taken from Tintinalli et al.'s Emergency Medicine: A Comprehensive Study Guide, 6 th edition textbook sections on MI and anaphylaxis (22).The objectives were clinically focused and specifically designed to include elements that would be necessary to successfully care for patients.PowerPoint (Microsoft Corporation, Redmond, WA) slides were created to use as the primary teaching adjunct to lecture sessions.Through an iterative and cooperative process, the advisory committee amended the objectives, altered the slides, and ultimately approved the learning materials.Learning objectives were identical for students receiving SIM and LEC training.
SIM case scenarios of critically ill MI and anaphylaxis patients were developed by the authors and reviewed by the advisory committee.Critical actions specific to the MI and anaphylaxis case were developed, and the individual actions were weighted by consensus, with higher point values assigned to actions deemed more critical.Components of the checklist were history, physical examination, diagnostic evaluation, and treatment.Each item was scored in a performed/not performed fashion.This checklist is similar to those used in simulation research (Appendixes 1, 2) (20).All lecture materials, case details, and assessment methods were then reviewed and approved by the EM clerkship director before implementation.Each student's performance was evaluated real-time during the SIM scenario.The majority of training and assessment was conducted by one of the authors (CEM).To ensure quality of the teaching sessions and reliability of the assessment tools, the EM clerkship director (SL) observed 25% of the sessions and reviewed students' scores.The EM clerkship director independently scored each student's performance during the SIM scenario observations and subsequently compared results with the faculty administering the SIM scenario.No discrepancies in scores were found, although a formal inter-rater reliability calculation was not performed.

Instruction Day 1
Students were randomized to begin instruction in the SIM-based training group or LEC group (Figure 1).The topic for instruction day 1 was MI.While the SIM group learned how to assess and manage a critically ill patient presenting with MI using SIM-based training, the LEC group learned this content via standard lecture with PowerPoint slides.Both teaching sessions were approximately 1 h.
After the teaching sessions were complete, each student, regardless of group, was individually evaluated on assessment and management of a critically ill patient presenting with MI during a SIM scenario.We chose to evaluate each student on a SIM scenario, as the advisory committee considered this to more closely reflect the goals of the training sessions: to manage a critical patient, not simply answer questions about managing a critical patient.The SIM scenarios were administered by faculty who operated the simulation software to react real-time to the student's treatment plans, provided the clinical case, and interacted with the student in the role of the patient and associated health care team.Each student's performance was scored according to the method detailed above.

Instruction Day 2
On instruction day 2, students switched groups.Students previously taught via LEC were now assigned to SIM based-training and vice versa (Figure 1).The topic for instruction day 2 was anaphylaxis.Training and evaluation were identical to day 1.This design allowed each student to serve as their own control.After the instruction sessions were complete, each student was individually evaluated on his or her assessment and management of a simulated critically ill patient presenting with anaphylaxis.Each student's performance was scored on a predetermined checklist specific for the anaphylaxis case scenario.
At the end of instruction day 2, each student had been evaluated on two separate topics, one they learned by SIMbased training and the other learned from didactic LEC with PowerPoint instruction.In addition, all students had an equal amount of exposure to the SIM before the intervention days, as well to both SIM-based and LEC training.

Statistical Analysis
At study end, each student had two performance checklists, one for a scenario after LEC, and the other after SIM.Scores were converted to a percentage of possible points, and compared between SIM and LEC groups.History, physical, and management component scores were calculated and converted to percentage scores.Means and 95% confidence intervals (CIs) were calculated.Each student's simulation score was compared with his/her lecture score.As the results were paired and non-parametric, we considered a signed-rank test with two-tailed alpha Ͻ 0.05 to represent the statistical difference between groups.

DISCUSSION
Our objective was to determine whether full-scale human patient SIM is superior to LEC for teaching medical students to assess and manage simulated critically ill patients.We found a significant improvement in student performance for those trained with the SIM compared with those trained via LEC.Twenty-seven of 28 students performed better on the assessment when trained on the SIM.The absolute improvement in scores for SIM-based training was 22%.We believe this is a meaningful improvement.
This study attempted to address some of the gaps in medical SIM literature.For example, Schwartz et al. reported that human patient SIM showed no significant difference in medical student performance (21).However, they noted that baseline academic achievement differences in their study groups occurred despite random assignment, and this may have affected their results   (21).By comparing each student's individual SIM score with their individual LEC score, we eliminate confounding due to baseline differences in knowledge, intellect, or test-taking skills.Schwartz et al. also noted that students studied on their own, outside of prescribed reading and didactics, and "a difference in study habits between the two intervention groups could potentially mask a difference in the efficacy of the educational modalities tested" (21).This study controlled for this issue, as all students had the same amount of "study time," which included only LEC and SIM time.There was no additional time for the students to study between instruction and testing.Therefore, with the same amount of time to prepare, the student's performance more accurately reflects the intervention, whether it was SIM or LEC.
Steadman et al. reported a randomized trial comparing SIM-based training to problem-based learning, and concluded that, for fourth-year medical students, SIM-based learning was superior to problem-based learning for the acquisition of critical assessment and management skills (20).The authors report that the use of SIM for the final assessment may have given the SIM group an advantage due to SIM bias, a bias introduced when the students have more experience on the SIM due to training on the SIM, thus increasing their comfort with the modality.In this study, each student had an equal amount of exposure to both the SIM and LEC teaching modalities, thus minimizing this potential bias.Further, SIM bias would predict that those students who had SIM training on day 1 would outperform those who had it on day 2.This was not observed in our study, as students performed better with SIM than with LEC, regardless of which method they received first.
In our study, the frequency and types of questions students asked were quite different between the SIM and LEC groups.For example, during the LEC training, typical questions were "What are the inferior leads on the ECG?" or "How many millimeters of ST change is important?"In the SIM-based training, students asked similar content questions but frequently added highly practical questions like "Do you give oxygen via mask or nasal cannula?How many liters?,""Which size IV should we use?," and "In what part of the body should I put the epinephrine?"(for anaphylaxis cases).These types of questions were never asked during the lecture sessions.The differences in questions suggest different learning pathways being activated and likely account for the improved scores.Aside from the high-fidelity human mannequin, the SIM scenario used real equipment such as laryngoscopes, intravenous tubing, endotracheal tubes, and cardiac monitors.We believe these physical stimuli allow the student to activate visual, auditory, and tactile learning pathways that a didactic lecture cannot.A structured literature review on highfidelity patient simulators for EM teaching concluded that the greatest impact of simulators would be to teach skills that cannot easily be taught any other way in a clinical setting, due to either their complexity or their rarity (23).
The delivery of medical education should be as diverse as the learning styles of the students consuming it, and should draw upon the strengths of all learning modalities.SIM-, LEC-, and problem/case-based learning have varying strengths and weaknesses.It is likely that the optimal way to deliver medical education is a combination of these teaching modalities.We must work to determine what information and skills are best taught by each of these modalities.This study found that SIMbased training was superior to LEC teaching for the assessment and management of critically ill MI and anaphylaxis patients.

Limitations
Our study has several limitations.We did not perform baseline assessments of the medical students on a SIM case before intervention.Thus, we are unable to determine the magnitude of change in baseline knowledge or performance provided by LEC or SIM training.However, literature already exists that shows that simulation can improve baseline scores (20,24).In our study, we sought only to answer whether LEC or SIM was more effective for training, which our data allow us to evaluate.
The LEC, SIM training sessions, and student assessments were performed by non-blinded physicians.We attempted to minimize this potential bias by creating an advisory committee consisting of six emergency physicians to guide and review the learning objectives, lecture, and assessment tools.Further, we used a standard checklist for scoring students' performance.Twenty-five percent of SIM evaluation sessions were scored simultaneously by another EM faculty member and there were no differences in scoring.
In our study, student evaluations were conducted during a SIM scenario.Would SIM training have been more effective than LEC had students been evaluated on a more traditional written examination?Our study cannot determine this.We considered that performance on a simulator more closely mirrors performance on an actual patient than do standard paper tests, and chose to test students according to this supposition.A primary goal of fourth-year medical education is to convert learned knowledge into patient-focused action.We argue that, aside from direct supervision of real cases (which is not safe for critical patients), assessment on a simulator would be preferred to determine competency of fourth-year students.We suspect that SIM-based training actually promotes the rapid translation of knowledge ("knows") into reasoned action ("does"), a leap of cognition that is poorly assessed by written testing (24).
We did not attempt to determine whether performance on the simulator translates into improved performance in real clinical settings.There are a few studies linking human patient SIM with actual patient outcomes.In a randomized controlled trial, Hall et al. found that when tested on intubating real patients in the operating room, paramedic students who were trained on a simulator were as effective as students trained on human subjects (25).In a retrospective case-control study, Wayne et al. reported that SIMtrained residents showed significantly higher adherence to American Heart Association standards when compared to traditionally trained residents during real Advanced Cardiac Life Support events in the hospital (26).These data suggest that performance on SIM may translate favorably to patient care.
Finally, as the testing occurred immediately after the teaching sessions, we cannot determine which teaching method leads to better long-term retention of knowledge and skills.

Figure 2 .
Figure 2. Comparison of overall score between simulation (SIM) and traditional didactic lecture (LEC) groups.

Figure 3 .
Figure 3.Comparison of scores between SIM and LEC groups by components: history, physical examination, and management.LEC(T) ‫؍‬ lecture; SIM ‫؍‬ simulation; CP ‫؍‬ chest pain (myocardial infarction) case; A ‫؍‬ anaphylaxis case.