Correlation of attending and patient assessment of resident communication skills in the emergency department

Communication and interpersonal skills are one of the Accreditation Council for Graduate Medical Education's six core competencies. Validated methods for assessing these among trainees are lacking. Educators have developed various communication assessment tools from both the supervising attending and the patient perspectives. How these different assessment methods and tools compare with each other remains unknown. The goal of this study was to determine the degree of agreement between attending and patient assessment of resident communication skills.

Monday through Friday. Patients were included if they could identify the resident who cared for them by photo; did not require interpreter services; and were at baseline alert and oriented to person, place, and time. Additionally, CAT surveys were administered to discharged patients only in accordance with our institution's patient survey policy. If eligible, the RA invited the patient to complete the CAT to provide valuable feedback on the resident physician's communication skills. The CAT, as described above, includes 15 questions based on a 1 to 5 Likert scale with 14 physician-specific questions for a total out of 70 possible points ( Figure 2). Our modified CAT included these 14 questions to obtain information specific to the individual resident and not the entire health care team. By completing the survey, patients gave consent for use of deidentified data for research purposes. Both attending physicians and patients filling out their respective assessments were made aware that their responses remain anonymous to the resident. Mean attending ratings for each resident across all three domains and mean CAT scores using the 14 physician-specific questions were calculated for each resident and compared for agreement.

Data analysis
Mean attending ratings and CAT scores were divided into tertiles due to nonparametric distribution of scores. Agreement between attending ratings and CAT scores of residents were measured using Cohen's kappa for each attending evaluation question. We used weighted scores to assign credit to partially concordant tertiles.
Tertiles that were in complete agreement were assigned a weight of 1, tertiles that was in partial agreement were assigned a weight of 0.5, and tertiles that were completely discordant were assigned a weight of 0. We a priori decided that a kappa less than 0 indicated poor agreement, a kappa between 0 and 0.2 represented slight agreement, a kappa between 0.2 and 0.4 represented fair agreement, a kappa between 0.4 and 0.6 represented moderate agreement, a kappa between 0.6 and 0.8 represented substantial agreement, and a kappa between 0.8 and 1 represented almost perfect agreement. 23 Given that the scale does have overlap between the categories, we chose the more conservative approach of choosing the lower agreement category.

RE SULTS
Twenty-six residents were evaluated during the study period. were missing for the domain of "communication with other physician colleagues and consultants" and one response was missing for "communication with nursing and ancillary staff" from the attending evaluations. Mean evaluation scores for all residents are provided in Figure 3. There was no difference between mean attending scores for PGY-1 and PGY-2. Mean scores for the individual CAT questions are displayed in Table 1. Patients rated the residents most highly on the CAT question of "treated me with respect," "let me talk without interruptions," and "talked in terms I could understand." They rated them the lowest on "encouraged me to ask questions." There was no difference between PGY-1 and PGY-2 across all CAT questions (Table 1). Tertile mean ranges for each scoring system are presented in Table 2. The range of mean CAT score associated with each tertile of attending evaluation question are also shown in Table 2

DISCUSS ION
The results of our study demonstrate statistically fair agreement between supervising attending physician and patient assessment of several different communication domains in EM residents. Given that the kappas of are just above the statistical threshold for fair, the clinical significance of this is closer to slight to fair agreement.
To the best of our knowledge, this is the first study to investigate degree of concordance in these two different assessor groups in EM residents and our results highlight the utility of using a multisource feedback approach to assessing resident communication and interpersonal skills. Even among faculty raters, prior work has shown marked variability and even poor agreement of assessing resident's skills. 24,25 Thus, the slight to fair agreement we found invites further research for educators and residency programs seeking to include patient feedback in resident evaluations in that it may provide additional critical information that may be missed by utilizing only attending evaluation methods.
The faculty attending and patient perspectives on residents' abilities are inherently different and most certainly account for lack of "strong" agreement. For one, these two groups typically spend different amounts of time in the actual bedside interaction.
While attending level direct observation has been noted to be a valid and well-received method of assessment, 9,11,26 the reality of clinical practice is that the time a supervising attending physician can devote to the direct observation of residents communicating with patients at the bedside is often limited. The patient, moreover, is involved in the entire interaction being evaluated. Previous work has shown that overall direct observation time in ED was less than 4% of total shift time. 27 Interestingly, the highest concordance was found between attending evaluation of communication with nursing staff and the CAT score. This may have been influenced by direct feedback provided by nurses to the attending physician rather than first-hand observation by the attending physician. Alternatively, it is possible that residents communicate differently with individuals not formally evaluating them, which may provide a more genuine representation of their true communication skills. This finding would be worth investigating further in a future study. Nevertheless, while supervising attending physician evaluation has historically been the primary mode of assessing resident interpersonal and communication skills, given the In contrast to the attending, the patient has the opportunity to assess the resident's communication through a different lens: as the direct recipient rather than as an observer. Not only may this lead to a more encompassing period of observation compared to a single direct observation session by a supervisor but it also allows for the patient's own emotional experience of the health care interaction to enhance or otherwise influence their assessment of the resident. Prior work has demonstrated that patient feedback collected through a validated and credible method can have a positive impact on medical performance, although feedback needs to be provided to residents in a manner that facilitates discussion and encourages an actionable behavior change if needed. 28 There is increasing data supporting the fact that multisource feedback is essential in assessing all resident skills and can improve professional practice. 29 Our findings highlight the potential benefit of using a multimodal

LI M ITATI O N S
This study has several limitations. We used two different assessment tools with different questions, which may limit comparison, particularly since the CAT score does not specifically ask about "communication with physician colleagues and consultants" or "nursing and ancillary staff." This was done because of the existing faculty evaluation forms that were already in place based on the need for EM Milestone-based assessment and integration