Standardized Patients to Assess Resident Interpersonal Communication Skills and Professional Values Milestones

It has been a challenge to assess communication and professional values Milestones in emergency medicine (EM) residents using standardized methods, as mandated by the Accreditation Council for Graduate Medical Education (ACGME). This paper outlines an innovative method of assessing these Milestones using an established instructional method. EM faculty mapped the communication and professional values Milestones to an existing communication and interpersonal skills scale. We identified six communication-focused scenarios: death notification; informed consent; medical non-compliance; medical error; treatment refusal; and advanced directives. In a pilot, 18 EM residents completed these six standardized patient (SP) encounters. Our experience suggests SP encounters can support standardized direct observation of residents’ achievement of ACGME Milestones. Further effort can be made to create a tailored, behaviorally-anchored tool that uses the Milestones as the conceptual framework.


INTRODUCTION
Although effective communication with patients is an integral part of the role of all physicians and has gained the spotlight over the last decade, there is no established standard on how it should be taught and assessed during traditional medical training. [1][2][3][4][5][6] The urgency to address this gap is evident, as the literature indicates that deficiencies in communication skills can lead to higher malpractice rates, patient dissatisfaction, and adverse patient outcomes. [7][8][9] The Accreditation Council for Graduate Medical Education (ACGME) endorsed "Interpersonal and Communication Skills" and "Professionalism" as two of the six core competencies. These competencies, and newly mandated ACGME Milestones, are challenging to assess in the clinical setting due to varying faculty frames of reference and the influence of factors external to resident performance. 10 Faculty may use themselves, other doctors, or patient outcomes as frames of reference when assessing residents. In addition, faculty report that they often use "gut feeling" or "gestalt" to translate their observations to numerical assessment scores. 10 Standardized patient (SP) encounters with validated tools are an established method of assessing learners 11 and may offer a more consistent way to assess residents. The literature supports a correlation between patient surveys and SP-based assessments of learners, 12 as well as the use of SP feedback for training and assessment of residents. [13][14][15][16] Using this established method can offer a more reliable assessment of these residency Milestones. This project aimed to pilot an innovative SP-based model to assess the interpersonal communication skills and professionalism Milestones of emergency medicine (EM) residents.

METHODS
In 2005, the University of Illinois-College of Medicine at Chicago (UIC-COM) Clinical Performance Center (CPC) developed an institution-based competency tool to provide resident performance data to program directors (PD). The Communication and Interpersonal Skills Objective Structured Clinical Exam (CIS-OSCE) was administered and analyzed across specialties including internal medicine, family medicine,

Standardized Patients to Assess Resident Interpersonal Communication Skills amd Professional Values
Vora et al.
surgery, pediatrics, neurology, and obstetrics-gynecology. 17 Later, a many-faceted Rasch measurement model was used to further analyze each item on the scale and the results of this analysis were used to create an improved communication rating scale. 18 We used the new Revised Communication and Interpersonal Skills (RUCIS) scale, a four-category behaviorally anchored rating scale (Table). Using a mapping method, four EM academic residency faculty integrated the Milestones into this existing RUCIS scale. Already being familiar with the Interpersonal and Communication Skills and Professionalism Milestones, the faculty members were given a chance to review the RUCIS Scale (with no modification to the anchors) and the details of the six communication tasks. They were then asked to individually map each of the behaviorally-anchored ratings on the RUCIS scale to a specific Milestone and level. Then the mapping was reviewed as a group and consensus was reached through iterative discussion until consensus was reached among all faculty ( Table  1). Eleven of the 13 RUCIS items were successfully mapped to specific levels of the two EM Milestones (Milestone 20 PROF1 and Milestone 22 ICS1). In the 11 items, Levels 1 through 4 of both Milestones were represented, with Milestone 20 measured five times and Milestone 22 measured six times.
In parallel to the Milestone mapping, the communication challenges were identified and developed. The six communication tasks were identified based on the previously developed patientcentered communication competency assessment implemented in the CPC in 2003. The tasks were originally identified based on the communication literature and their salience to clinical practice. As noted by the authors, "they were designed to allow residents to demonstrate their skills across a range of patient ages, genders, and problems". 9 For our Milestone assessment initiative, the tasks used were the following: giving bad news; obtaining informed consent; patient education (addressing medication non-compliance); medical error; treatment refusal; and advanced directives. The cases for each of these tasks were either adapted to the EM setting from previous cases used by other specialties or were newly created and validated by EM faculty through iterative review. Each case was designed to present a communication task with an underlying communication challenge. For example, Milestone: level 3. Listening to my story ( ) You rarely gave me any opportunity to tell my story and/or frequently interrupted me while I was talking, not allowing me to finish what I was saying. Sometimes I felt you were not paying attention (for example, you asked for information that I already provided).  the communication task in the "giving bad news" case was for the resident to appropriately deliver a death notification and the communication challenge was for the resident to address the need for an autopsy of the deceased. Each SP encounter was 10 minutes, with the SP completing the RUCIS scale immediately following the encounter. This was followed by 10 minutes for SP-to-resident debriefing. This study was included under the Clinical Performance Center Institutional Review Board (or human subjects committee) approval.

RESULTS
As this was intended as a pilot of an innovative Milestone assessment method, the sample size was small and collected data was limited. An individualized score report was provided to the CCC for each resident that included the Milestone score for each of the two Milestones. See Table 2 as an example. The score report was included for faculty to review as part of the resident's  file; but as this was a pilot, it was not incorporated in any specific numerical way into the resident's overall Milestone score.
Additionally, in the individual survey 94% of residents agreed that verbal feedback from the SP was helpful and 100% of residents felt the cases allowed them to demonstrate their communication and professionalism skills. In the faculty debriefing, residents uniformly agreed the SP encounter and feedback would improve the quality of care for future patients.

DISCUSSION
In this program, essential communication skills were assessed and EM residents received feedback from the SP as well as an EM faculty member in a simulated setting. This paper demonstrates the utilization of an established OSCE method for Milestone assessment that could provide useful, quantitative performance data to a residency CCC. Although the CCC did not use this pilot data in a structured way, there is potential for standardized incorporation of these scores in the future. Using a larger sample size, it would be important to look at the correlation of individual resident OSCE scores with other assessments in their file. 19 Other possibilities would be to correlate resident scores to level of experience and comfort. Although in this pilot variability in resident scores across experience level was noted, the sample size was too small to report any statistically significant correlation data. Also, SPs were able to offer targeted feedback to individual learners and a difficult-to-obtain patient perspective. This method of assessment is reported in the literature, 13,14 but further study is needed to assess resident communication skills Milestone improvement after SP debriefing.
In this pilot, a previously created tool was mapped to the Milestones. Since patient-centered care was the conceptual framework for the RUCIS scale and not the ACGME EM Milestones, it is necessarily limited in directly assessing ACGME EM Milestones. Further work is needed to create a new, targeted assessment tool that can be used in conjunction with the established OSCE methodology to specifically assess interpersonal communication and professional values Milestones. Using the Milestones as a conceptual framework, a behaviorally anchored tool could be created, similar to the CIS-OSCE, to assess specific behaviors as outlined by the ACGME Milestones. This type of tool could provide consistent, reliable, quantitative data to residency PDs and enhance the instruction and assessment of residents throughout their training with the ultimate goal of improving these skills in patient interactions.

LIMITATIONS
As this was a pilot study, there was an anticipated limitation in sufficient data collection to perform definitive quantitative analyses. Residents knew they were scheduled for communication OSCE encounters and may have focused on demonstrating strong communication. Thus, like many standardized assessments these can be best thought of as "maximal performance" assessments, which may not reflect

Standardized Patients to Assess Resident Interpersonal Communication Skills amd Professional Values
Vora et al.
learners' typical performance or "worst-case" performance. 20 Also, as with many OSCEs, it is possible that residents who were scheduled earlier discussed some case content with laterscheduled residents although the scores of later residents did not reflect this to be the case.
Of note, when the Milestones were mapped to the RUCIS scale, level 5 of both Milestones was not represented in the current tool, although it is notable that residents are not expected to reach level 5 during residency training. This may speak to the need to develop specific assessment methods to measure higher level Milestones with OSCE assessments used for early level Milestones. In addition, the scores provided to the CCC reported a number as a continuous variable as opposed to an ordinal variable as required by the Milestone scale, which may have limited their usefulness. In the future, qualitative feedback from the PD or the CCC on the value of the mapped OSCE score could inform score report structure. This program was piloted at one institution, which would limit its generalizability. Due to these limitations, it would be worthwhile to explore creating a new assessment tool with the EM Milestones as the underlying conceptual framework.

CONCLUSION
As competency-based medical education has come to the forefront, there is a need for reliable and valid methods of assessing communication and professionalism skills. This pilot supports the potential use of an established method to conduct a more rigorous assessment of interpersonal communication skills and professional values Milestones of EM residents. Future studies may also compare SP assessment to standard simulation assessment of these skills to further expand the Milestone assessment toolbox.