High‐efficiency Practices of Residents in an Academic Emergency Department: A Mixed‐methods Study

Emergency department utilization and crowding is increasing, putting additional pressure on emergency medicine (EM) residency programs to train efficient residents who can meet these demands. Specific practices associated with resident efficiency have yet to be identified. The objective of this study was to identify practices associated with enhanced efficiency in EM residents.

and is seen by many as a public health crisis, an issue compounded by projected physician shortages. 1,5 Emergency medicine (EM) training programs are therefore under increased pressure to produce residents who are equipped with the skills and practices needed to face these challenges when working in independent clinical practice. Therefore, an enhanced understanding of factors that contribute to individual efficiency in residents would have widespread implications for both EM education and patient care.
Previous research has demonstrated that resident productivity increases overall ED efficiency, [6][7][8] and that resident efficiency increases throughout training. [9][10][11] As over half of EM residents go on to work in community EDs after graduation, 12 it is crucial for residency programs to equip graduates with specific efficiency skills that can be relied upon during independent practice. A recently published study analyzing community EM providers identified several specific behaviors that were independently associated with provider efficiency. 13 Obtaining similar data from EM residents would not only provide residents with specific skills to improve their efficiency during training but would also allow for comparison of efficiency behaviors utilized by EM providers and residents. This information could provide direction and a framework for incorporating evidence-based efficiency education into residency programs. The objective of this study was to identify key practices associated with enhanced efficiency in EM residents through observation in a large academic ED.

Study Design
This was a mixed-methods study utilizing semistructured qualitative interviews to elicit specific behaviors that were then used to guide an observational study of ED residents. The study was approved by the local institutional review board with a waiver of documentation of informed consent, and verbal consent was obtained.

Study Setting and Population
Observational data were collected between May and July 2019 from a midwestern university ED with an annual volume of 56,000 visits. Participants in the qualitative stage of the study were EM faculty who were regularly involved in the resident education program (e.g., residency program leadership, members of departmental education committee). Participants in the quantitative observational stage of the study were resident physicians and resident advanced practice providers (i.e., physician assistants and nurse practitioners). No compensation was provided and there was no follow-up for any study participants.

Qualitative Stage
Semistructured qualitative interviews were conducted to produce a comprehensive list of behaviors thought to be associated with resident efficiency. Eight participants were interviewed by phone, e-mail, or face to face and asked to answer the following question: "What behaviors of emergency medicine residents do you think are associated with efficiency?" Interviews were conducted by a single research assistant, and participants were encouraged to generate a comprehensive list of behaviors, but there was no further probing of participants. Responses were recorded in a list of potential efficiency behaviors. The list of behaviors was reviewed independently by two members of the study team, and each item on the list was coded to a defined theme. Items determined to have duplicate content were mapped to a single theme. Discrepancies were settled by consensus of the research team until theme saturation was reached and a comprehensive list of themes was produced. Four reviewers, from the same academic medical center, then rated the themes based on predetermined criteria. The roles of the reviewers included academic EM physician, ED registered nurse, EM PGY-3 resident, and a medical student. A researcher met with each reviewer to provide training on the rating process. Reviewers then rated each theme on three criteria: 1) measurability: a researcher can measure this behavior by observing an ED resident; 2) teachability: a resident can change this behavior if he/she chooses; and 3) relationship to efficiency: this behavior is associated with efficiency, either positively or negatively. Raters assessed each theme on a Likert scale of 1 (strongly disagree) to 5 (strongly agree) for each of the three criteria. Ratings for each criterion were averaged across the four independent rankings, and each theme was given an impact score based on the sum of the three averaged criteria. Themes in the top 50th percentile based on impact score were included. These behaviors were combined with a previously used list of 16 candidate efficiency behaviors, 13 and duplicates were discarded. Eight novel themes from the resident-specific qualitative interviews were added to the previous list, 13 plus an additional nine themes generated from study team consensus. The study team included two EM physicians with leadership roles in graduate medical education and a research scientist with expertise in measurement and mixed methods. This created a final list of 33 efficiency behaviors (Table 1 and Figure 1).
Observation Stage Participant Selection. To be included in this study, residents needed to be scheduled for two separate shifts in the ED between May and July 2019 and must not have participated in the qualitative stage of this study.
Protocol. For the observational stage of the study, EM residents provided verbal consent prior to the observations. EM residents were observed over two independent 4-hour sessions, for 8 total hours of observation. The observations were scheduled during historically high-volume hours in the ED, between 0900 and 2200. A single trained researcher recorded minute-by-minute written observation logs to capture the timing and frequency of each of the 33 behaviors identified in the qualitative stage of the study. The researcher recording behaviors was blinded to the outcome: resident efficiency. When multiple activities occurred in the same minute, each activity was given equal parts of the minute. 14 Participants were instructed to perform normal clinical activities, and any time the participants spent interacting with the researcher was excluded from the total observation time for analysis.
Measures. Efficiency was quantified using relative value units per hour (RVU/hour), 15 and a sensitivity analysis was conducted using patients per hour. The RVU/hour measure incorporates both patient volume and severity of illness. 16 Although RVU/hour is determined for each patient based on faculty billing activity, it is also used as a standard measurement to evaluate resident productivity by medical directors and residency programs. 17 RVU/hour was obtained from ED billing data and averaged over 6 months and 12 months for each subject. Resident advanced practice providers (i.e., physician assistants and nurse practitioners) were included as their residency practice patterns are identical to that of the resident physicians in our center. For the data analysis, the resident advanced practice providers in their first and second years of training were included with PGY-1 and PGY-2 physician residents, respectively. Data Analysis. Descriptive statistics (median and mean) of each behavior were used to present the underlying distributions of the behaviors among study participants. Generalized estimating equations (distribution = normal; link = identity) clustered on resident were then used to estimate the association between behavior and efficiency outcome. First, univariable models were estimated for the primary outcome of 6-month RVU/hour. A multivariable model was then estimated to assess for confounding by training year. Confounding by training year was considered if the coefficient changed by over 10%, and the confounder was associated with behavior. Previous studies had identified that training year is associated with efficiency, so it was hypothesized to be a confounder. [9][10][11] Multivariable models, adjusted for training year (PGY) were constructed for the primary outcome. Statistical analysis was conducted in SAS (version 9.4, SAS Institute, Cary, NC).
Sensitivity Analyses. Medical trainees often have a learning curve related to efficiency, 9 so a 12-month average of RVU/hour was used to assess the robustness of study results to a longer outcome ascertainment period. Further, RVU/hour may not fully capture efficiency, because RVUs combine the number of patients seen and the intensity of care provided to these patients. A second sensitivity analysis was conducted using patients per hour as an outcome.

Description of Study Population and Behaviors
Qualitative Phase. In addition to previously demonstrated behaviors, new behaviors (n = 61) were elicited from open-ended interview questions about resident-specific efficiency behaviors, and 26 were removed as duplicates ( Figure 1). Through independent ranking of the themes, 27 themes were excluded based on impact score (median impact score = 12.0 of 15 total, interquartile range [IQR] = 10.9-13.3). The remaining eight themes, in addition to the 16 behaviors previously studied in EM efficiency 13 and nine behaviors from the study team, were included in the current study ( Figure 1 and Table 1). Quantitative Phase. There were 32 residents eligible for participation in the study. Four residents did not have two separate EM shifts during the study data collection period, and one resident declined to participate in the study (96% consent rate; Figure 1). There were 27 participants in the study; 85% were physician residents (PGY-3 n = 8; PGY-2 n = 7; PGY-1 n = 8) and 15% were resident advanced practice providers (physician assistant or nurse practitioner). Median and average frequencies and times of behaviors are listed in Data Supplement S1, Table S1 (, available as supporting information in the online version of this paper, which is available at http://onlinelibrary.wi ley.com/doi/10.1002/aet2.10517/full) Some behaviors varied by training year (PGY), including conversations with staff, visits to patient room, and EMR review; resident efficiency (RVUs/hr) were also positively associated with increasing PGY (Figure 2).
Main Results. The median (IQR) 6-month RVUs/hr were 3.2 (2.4-3.6). In unadjusted models, 14 behaviors were associated with the primary outcome of 6-month RVUs/hr ( Table 2). After training year was adjusted for, 10 behaviors were associated with efficiency (Table 2). Seven behaviors were associated with increased efficiency: taking initial patient history with the nurse present (number/hour and per new patient), utilizing dictation (number/hour and % time), running the board (number/hour), conversations with other care team members (number/hour and % time), smartphone text conversation (number/ hour and % time), nonwork tasks (number/hour), and average patient load. Per hour, each additional instance of taking history with a nurse present was associated with a 0.71 increase in RVU/hour (95% CI = 0.19 to 1.22), dictating notes was associated with a 0.26 increase in RVU/hour (95% confidence interval [CI] = 0.01 to 0.51), running the board was associated with a 0.16 increase in RVU/hour (95% CI = 0.04 to 0.28), having a conversation with other care team members was associated with a 0.05 increase in RVU/hour (95% CI = 0.01 to 0.09), having a smartphone text conversation was associated with a 0.23 increase in RVU/hour (95% CI = 0.07 to 0.38), and engaging in a nonwork tasks was    Figure 1. Flowchart of study design. *Four residents were not scheduled for any shifts in the ED during the data collection timeline and were therefore excluded from the study. GEE = generalized estimating equations; RVU = relative value units.
AEM EDUCATION AND TRAINING • www.aem-e-t.com Sensitivity Analyses. Robustness of results was evaluated with sensitivity analyses of a longer efficiency sample (12-month mean RVU/hour) and a different efficiency outcome, patients per hour. Similar behaviors were identified as associated with efficiency using a 12-month RVU/hour average (Data Supplement S1, Table S2). Using 6-month averages of patients per hour as the efficiency measure, results were again similar except for the new behavior of entering orders while in the patient's room being associated with increased efficiency (% time, b = 0.34, 95% CI = 0.03 to 0.65; Data Supplement S1, Table S3). An additional analysis was conducted with only the physician residents, and results were similar (Data Supplement S1, Table S4). The additional efficiency behavior of entering initial orders in the patient room was associated with increased efficiency (% time, b = 1.14, 95% CI = 0.19 to 2.08).

DISCUSSION
The results of our study reflect the individual variability in resident efficiency. Our study focused on teachable and measurable behaviors in trainees. As noted with previous studies, resident efficiency increases as a learner progresses through training, with the largest differences seen in the first year of training. [9][10][11] Our study demonstrated several behaviors that correlate positively with resident efficiency. The strongest behaviors were the percentage of time note dictation was used and the percentage of time that communication with the care team occurred via smartphone texting. The strongest behavior that negatively impacted efficiency was number of visits to patient's rooms, which decreased throughout training years. While this study is the first to investigate resident behaviors associated with efficiency, a previous study had identified efficiency behaviors of community ED providers. 13 Three efficiency behaviors (average patient load, conversations with health care team, and running the board) were associated with increased efficiency for both residents and community providers. However, there were some differences for resident efficiency. First, nonwork tasks were associated with increased efficiency in this study of residents, whereas it was previously associated with decreased efficiency among community providers. This finding could be due to reverse causality (i.e., the more efficient residents take less time to complete their tasks and therefore have more time to engage in nonwork tasks). Further, visits to the patient room was a negative efficiency behavior in this study, but a positive efficiency behavior previously. These differences could be due to the addition of teaching tasks for residents, the differential resident workflow with the inclusion of medical students and attending physicians, or the variations in community and academic practice. The differences in visits to patient room could also be a product of selection bias, with learners opting to take on sicker patients who require more time and visits to the room. Finally, there were some behaviors that were only associated with efficiency among residents (taking history with nurse and EMR review) or only associated with efficiency among community providers (using team member names and documentation on previous patients). These differences could be due to different training/experience levels, different patient populations (i.e., potentially more medically complex patients in the academic ED), or differences in resident and community providers' workflows.
One easily implemented efficiency behavior is having residents take the patient history with the nurse in the room. In our team's observations, residents who arrived at the patient room quickly, while the nurse was still in the room, were able to begin a focused history and physical examination while the nurse simultaneously performs the intake evaluation and collects vitals. The residents were then able to inform both the patient and the nurse of the care plan and orders at the same time. This behavior was defined by a recently published article as "swarming" (quick evaluation of a patient simultaneously by the nurse, resident, and attending physician). The article found that swarming in an academic ED improves both departmental efficiency and resident education. 18 Interestingly, we observed that this behavior was rarely initiated by the resident themselves and generally occurred only when residents saw the patient quickly upon rooming. We suggest that residents and academic departments might consider actively prioritizing this behavior as a way of improving efficiency and communication between health care team members.
Another behavior found to be associated with enhanced efficiency is communicating with the care team via a secure smartphone text messaging system used by ED personnel at the study site. This communication system allows for residents to quickly send and receive messages and can save time they might have spent searching the department for members of the care team, thus increasing their efficiency. Alternately, it is possible that more efficient residents simply utilize smartphone texting more than their less efficient peers. A previous study on the implementation of a smartphone-enabled communication system on general internal medicine wards found that the system was highly utilized and was perceived to improve efficiency, accountability, timeliness, and collaboration by both physicians and nurses. 19 Previous literature has found the use of smartphone texting in patient care to be related to a higher rate of interruptions, 20 which has been predicted to lead to decreased efficiency. However, in this study, the number of smartphone interruptions were not found to have a significant effect on resident efficiency. Departmentwide adoption of smartphone messaging systems may be a way to improve efficiency on a systems level.
In this study, we evaluated objective and measurable findings that play a role in resident efficiency. As residents progress through training, in addition to developing core competencies and enhancing medical knowledge, a significant focus should be placed on their patterns and personal development of efficiency. Ultimately, it would be beneficial for residency programs to evaluate how to effectively teach and modify these behaviors to implement methods to help residents succeed as they enter independent practice. From previous studies, it appears that residents have the most development in between the first and second years of training, [9][10][11] suggesting that the time to intervene on pattern development is early in resident development.

LIMITATIONS
There are several important limitations in this study. First, the observational study design utilized allows only for the identification of association between specific behaviors and efficiency, not causation. It also introduces a potential Hawthorne effect, where EM resident participants may have altered their behavior in the presence of the observer. In an attempt to limit this, residents were observed on two separate shifts at different times. Additionally, residents were compared to other residents in their PGY who were also being observed. Second, provider efficiency in the ED is intrinsically difficult to quantify given the fluctuating and unpredictable workflow and patient volume. 15,21 To minimize these potential effects, observations were performed during peak shift hours based on historic patient volume and individual providers were observed across multiple shifts. Third, this study was conducted in a single academic ED in the Midwest. This may limit its external generalizability to other institutions. Although we have a small sample size, this study demonstrates changes at a systems level that can enhance efficiency such as providing dictation devices and availability of methods to communicate directly with ancillary staff, which both showed positive correlations with efficiency. Fourth, RVUs as a measure of provider efficiency are influenced by patient acuity and coding practices and has been accepted as a reasonable measure in previous studies. 17,21-23 However, we conducted a sensitivity analysis using patients per hour to provide another accepted measure of provider efficiency, which does not incorporate patient acuity. Finally, this study was intentionally limited to provider efficiency, not quality, and higher RVU/hour may not be associated with higher quality measures. Future research of additional measures of efficiency or the association of these efficiency behaviors with quality measures would be valuable.

CONCLUSIONS
This study identified key practices associated with enhanced efficiency in emergency medicine residents. The results of this study can be utilized by both residents and training programs. Residents can use these results to prompt self-assessment and to discover areas for improvement in their own efficiency. Some of the behaviors listed can be implemented immediately, through individual residents' initiative. Additionally, these results can guide residency faculty who are responsible for providing concrete feedback to residents. The relative value units per hour data commonly used in resident evaluations provides residents with a measure of their efficiency, but it does not suggest specific, evidence-based behaviors residents can implement to improve upon that efficiency. Faculty could use these results in evaluations to prompt discussion and to help residents develop a stronger sense of their existing strengths as well as set tangible, measurable goals for further growth. Ultimately, training programs may implement these changes at a systems level to improve resident efficiency education as well as overall efficiency of the department.