Emergency Medicine Residents Consistently Rate Themselves Higher than Attending Assessments on ACGME Milestones

Introduction In 2012 the Accreditation Council for Graduate Medical Education (ACGME) introduced the Next Accreditation System (NAS), which implemented milestones to assess the competency of residents and fellows. While attending evaluation and feedback is crucial for resident development, perhaps equally important is a resident’s self-assessment. If a resident does not accurately self-assess, clinical and professional progress may be compromised. The objective of our study was to compare emergency medicine (EM) resident milestone evaluation by EM faculty with the same resident’s self-assessment. Methods This is an observational, cross-sectional study that was performed at an academic, four-year EM residency program. Twenty-five randomly chosen residents completed milestone self-assessment using eight ACGME sub-competencies deemed by residency leadership as representative of core EM principles. These residents were also evaluated by 20 faculty members. The milestone levels were evaluated on a nine-point scale. We calculated the average difference between resident self-ratings and faculty ratings, and used sample t-tests to determine statistical significance of the difference in scores. Results Eighteen residents evaluated themselves. Each resident was assessed by an average of 16 attendings (min=10, max=20). Residents gave themselves statistically significant higher milestone ratings than attendings did for each sub-competency examined (p<0.0001). Conclusion Residents over-estimated their abilities in every sub-competency assessed. This underscores the importance of feedback and assessment transparency. More attention needs to be paid to methods by which residency leadership can make residents’ self-perception of their clinical ability more congruent with that of their teachers and evaluators. The major limitation of our study is small sample size of both residents and attendings.


INTRODUCTION
In 2012 the Accreditation Council for Graduate Medical Education (ACGME) introduced the Next Accreditation System (NAS), which implemented milestones to assess the abilities and progress of residents. Each milestone is a significant, progressive, competencybased point in the development of a resident. These milestones evaluate accomplishments that identify Yale University School of Medicine, Department of Emergency Medicine, New Haven, Connecticut specialty-specific knowledge, skills, attitudes and behaviors that can be used as outcome measures within the general competencies. 1,2 Emergency medicine (EM) has developed 23 sub-competencies, with five milestone levels within each. Residents are expected to progress through levels of proficiency as they complete their training. 2,3 Attending evaluation and feedback is crucial for resident development. However, at least as equally important is Residents Rate Themselves Higher than Attendings Goldflam et al.
a resident's self-assessment. This is because feedback from others is often interpreted and integrated through the framework of a learner's self-assessment. 4,5 Learners use an amalgam of self-assessment and feedback to generate actionable goals for improvement. 6 If a resident does not accurately self-assess, clinical and professional progress may be compromised. A resident who is unable to accurately judge his or her own abilities may fail to achieve the necessary skills to be a safe and effective physician. In other words, failure to acknowledge deficiencies may lead to a failure to correct them.
Attending physicians working at academic centers are used as the benchmark in assessing a resident's abilities as a physician. To date, no study has compared resident selfassessment to attending assessment using the standardized framework of the ACGME milestones.

Goals of this Investigation
Our study used the framework of the ACGME milestones to compare EM resident evaluation by EM faculty with the same residents' self-assessments.

METHODS
This study is an observational, cross-sectional study performed at an academic EM residency. A human investigation committee (HIC) exemption was granted by the institutional IRB. All residents from EM post graduate year (PGY) 1 through 4 level were included in the study, with the exception of the single resident who helped to conduct the study. Twenty-five residents were chosen using a random number generator to participate in the study. The remaining residents were omitted due to time limitations on attendings filling out the forms and concerns that too large a number of evaluations would be prohibitive to attending willingness to participate in the study.
These residents completed self-assessments of milestone levels using eight ACGME sub-competencies that were chosen as representative of core EM principles by residency leadership consensus. Moreover, residency leadership agreed that a large group of attending evaluators would likely be able to comment on these, more familiar, sub-competencies for the majority of residents. The residency leadership consensus consisted of the residency program director and associate program directors. These included Emergency Stabilization (PC1), History and Physical (PC2), Diagnostic Studies (PC3), Diagnosis (PC4), Disposition (PC7), Communication (ICS1), Multi-Tasking (PC8), and Team Management (ICS2). These same residents were also evaluated by 20 faculty members using identical milestones. Faculty members have contact with residents in various settings, which include clinical shifts, simulation laboratory, and in small-group teaching sessions. Faculty members were able to opt out of assessing any resident whom they felt they could not evaluate due to limited interaction. The sub-competencies were evaluated on a nine-point scale, which reflects the rubric published by the ACGME (Figure).
No advanced training or instruction was provided regarding the utilization of the ACGME milestones. No other evaluation tools were provided to faculty when they were asked to assign a score.
We calculated the average difference between resident self-ratings and faculty ratings. Sample t-tests were used to determine the statistical significance of the difference in scores. We carried out mixed models analyses to determine if there were any significant interactions between the rater type (self vs. attending) and program year. For each program year, we calculated and compared the difference in the least square means between residents and their attending raters to the overall difference in least square means for each sub-competency.

RESULTS
Eighteen of the 25 residents surveyed completed the evaluation. Each resident was assessed by an average of 16 attendings (min=10, max=20). Residents gave themselves higher milestone ratings than attendings did for each of the eight sub-competencies evaluated ( Table 1). The mean difference in score for each sub-competency was close to one point, with the exception of "Team Management," which was 0.5 points. For seven out of eight sub-competencies, the difference in resident milestone self-assessment score and attending milestone assessment score was statistically significant (p<0.05). The one sub-competency where statistical significance was not reached was "Team Management" (p=0.09).
Mixed model analysis showed statistically significant differences between self-ratings and attending ratings in most sub-competencies for the PGY 1 and 3 cohorts (Table 2 and  Table 3). The PGY 2 cohort had fewer differences across sub-competencies, with statistically significant differences in only three sub-competencies (Table 4). For PGY 4, self and attending ratings did not significantly differ in any subcompetency (Table 5).

DISCUSSION
Our study found that residents (combined PGY1 through PGY4) consistently rated themselves as more proficient for each sub-competency than did their attending evaluators. This is consistent with prior data showing that physician selfassessment typically does not correlate with external measures of performance. 7 Although self-assessment may be inaccurate, it is important for evaluators to consider learner self-image when giving feedback. This feedback will undoubtedly be interpreted by the learner through a filter of his/her own perception. 8 For example, feedback from an attending that is lower than a learner feels he/she attained, may be rejected by that learner who believes he or she has reached a higher level of proficiency. This could negatively impact the development and growth of that learner.
Our study illustrates that milestone-based assessment  converge and are quite similar. It is important to note that these p values were not adjusted for multiple comparisons and should therefore be interpreted as only part of further exploratory analyses.
Taking the results of our study into consideration, the finding that residents perceive themselves as more capable than they are rated by attendings would be relevant to discussions in Clinical Competency Committee (CCC) meetings. Residents' perception of their skills would be important in grading them on sub-competencies that deal with "practice-based learning and improvement." Although difficult to put into practice, perhaps resident self-evaluations should be included in their "residency portfolios" and compared to the CCC rating of that resident to ensure that as the resident moves through the program self-perception is not significantly different from that of his/her evaluators.

LIMITATIONS
The major limitation of this study is the small sample size of attendings and residents evaluated. Our self-assessment response rate was 70%. Self-assessment was not compulsory, as participation in research was voluntary per our HIC. It is not known if those who did not respond were different demographically or in PGY year, as the study personnel was blinded to the identities of the residents assessed. Another limitation is the varying levels of familiarity with the milestones among the residents and attendings surveyed. In addition, due to the nature of EM shift work, attendings have different frequencies of interactions with residents, which may introduce bias into their assessments. As the study was done Goldflam et al.
Residents Rate Themselves Higher than Attendings at an academic institution, some faculty members do have less clinical time in the department than others. Although faculty members were permitted to opt out of assessing a resident with whom they had limited experience, we acknowledge that the frequency and types of faculty-resident interactions assessed may vary widely. In addition, the residents studied were at different levels of their training; this may have influenced their self-ratings. For example, some of the residents surveyed were close to graduation, a circumstance that may inflate their selfassessments. Although our data suggest that PGY 4 residents' and attending evaluations may be similar, our interpretation is limited by the small number of representatives in each class; thus, more investigation is required to determine if there is a difference between classes in their ability to self-assess accurately. A larger sample of residents assessed may allow for more detailed sub-group analysis by PGY year. In addition, a larger sample size would also allow for more detailed analysis of high and low performers and their ability to selfassess, as had been demonstrated in the past. This study relies on the assumption that attending ratings are more accurate than resident self-rating, the validity of which may need further investigation. 4,7 Perhaps most importantly, milestones are a relatively new assessment tool with very few studies evaluating their validity. 1,3,9

CONCLUSION
Residents over-estimated their abilities in each of eight sub-competencies assessed. This underscores the importance of feedback and assessment transparency. More attention needs to be paid to methods by which residency leadership can make residents' clinical ability self-perception more congruent with that of their teachers and evaluators. Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. The authors disclosed none.