A Competency-based Tool for Resident Evaluation of Pediatric Emergency Department Faculty

<jats:p>n/a</jats:p>


BACKGROUND
The quality of teaching skills among faculty is critical to allowing trainees to gain competence for independent practice, and while considerable progress has been made on trainee assessment, faculty evaluation tools have lagged behind. [1][2][3] The Accreditation Council for Graduate Medical Education (ACGME) Common Program Requirements dictate that residents must evaluate faculty, underscoring the need for quality evaluation tools. 4,5 A competency-based evaluation (CBE) emphasizes behaviorally focused skills and developmental outcomes, and has long been used to assess trainees; however, there are no published CBEs for faculty in pediatric emergency medicine (PEM), and limited tools in pediatrics and EM. [6][7][8][9] The ACGME has established six core physician competencies that are the gold standard in graduate medical education and continue to be developed into specialty-specific CBEs with "Pediatric Milestones 2.0" and "Emergency Medicine Milestones 2.0." 4,5,[10][11][12] To address the lack of faculty assessments in PEM, we aimed to develop a specific CBE using the conceptual framework of the ACGME milestones, with behavioral anchors incorporating previously published tools, such as the Stanford Faculty Development Program. [13][14][15][16]

OBJECTIVES
Our objectives were to develop a CBE tool 1) for formative assessment on pediatric emergency department (PED)-specific teaching skills, including procedural instruction, 2) that trainees perceive as efficient and effective, and 3) that faculty find useful for their development as educators.

CURRICULAR DESIGN
The CBE was designed using iterative review by a cohort of six clinician-educators from the departments of pediatrics and emergency medicine, including graduates of the Harvard Macy Institute, American College of Emergency Physicians teaching fellowship, and the Johns Hopkins University Master of Education in the Health Professions. The cohort members all have experience on clinical competency committees, and in faculty and program evaluation. For content validity, we conducted a literature review of existing tools, and through consensus methodology we identified skills critical to a PED attending and developed sub-competencies from the ACGME core competencies. Behavioral anchors were adapted from a CBE used for general surgery faculty, the Pediatric and Emergency Medicine Milestones, and the Stanford Faculty Development tool. 2,4,5,14 We used a milestone scale, with half-steps to indicate that the lower milestone has been demonstrated, as well as some skills of the higher milestone. The tool was then evaluated by additional educational reviewers beyond the initial cohort to ensure efficiency of use and readability, and to consider whether critical items had not yet been included. The review process resulted in 11 sub-competencies distributed over the ACGME competencies (Table).
Trainees completed both an existing Likert assessment without behavioral anchors and the CBEs over a six-month  Attempts to identify unique aspects of each patient and use them to establish an effective physician-patient alliance. Approaches all healthcare professionals in the same way, regardless of their role in patient care.
Systematically identifies the unique needs of each patient and uses them to build a strong physician-patient relationship.
Effectively communicates with other healthcare professionals with an understanding of their role in patient care.
Effortlessly identifies the unique needs of each patient and builds an authentic relationship with them and their support system. Seamlessly broaches sensitive topics in a way that puts patients at ease. Approaches other healthcare professionals as individuals to build a working relationship that provides the best outcomes for the patient. Encourages residents to share opinions and provide individualized teaching based on resident competency level. Provides the tools and motivation necessary for residents to formulate essential questions and to self-teach complex topics.
PED, pediatric emergency department; EBM, evidence-based medicine period. Trainees and faculty were surveyed on use of the CBE tool for efficacy, efficiency, and satisfaction to determine Kirkpatrick Level 1 (reactions) outcome attainment at the conclusion of the pilot period. 17 No faculty or residents who completed the surveys were involved in the development of the tool. This initiative was deemed exempt by the University of Maryland Institutional Review Board.

IMPACT/EFFECTIVENESS
A total of 143 CBEs of seven PED faculty were assigned, and trainees completed all assigned evaluations. All faculty (7), and 45% of residents (17) responded to the survey; survey items were rated on a five-point scale. Primarily pediatrics trainees completed the survey (10) and included both intern (10) and senior (7) trainees. The CBE tool was rated by 71% of residents as easy to understand (mean 3.6, SD 0.6), and 76% agreed or strongly agreed the CBE allowed them to effectively evaluate faculty (mean 3.9, SD 0.6). Most residents agreed or strongly agreed they are satisfied with the CBE (mean 3.8, SD 0.6), with no residents disagreeing. After reviewing six months of their CBEs, 71% of faculty reported the tool was formative (mean 4.3, SD 1), and 86% felt it was easy to understand (mean 4.4, SD 0.8). Importantly, 86% of faculty agreed with the areas for improvement identified (mean 4.4, SD 0.8).
The CBE was longer than the previous Likert evaluation; however, trainees felt the CBE remained efficient. The milestone scales with behavioral anchors and halfsteps were intended to indicate when faculty were between milestones. While this mirrors the design of the ACGME milestones, it may have created the opportunity for personal bias to affect the CBE. 4,5 Additional limitations include our use of a small sample size of faculty, trainees from multiple specialties with possible different expectations of faculty, and a lower response rate on trainee surveys. Importantly, our CBE tool introduced two subcompetency items addressing technical skills and procedural autonomy. The use of procedural-focused faculty competencies is unique, with only two previously published items, one within general surgery, and one within an EM shift-based feedback tool. 2,6 Procedures are a significant component of PEM and a critical area of assessment for faculty. Procedural autonomy is of critical importance to trainees' development; however, we acknowledge that a trainee's opinion of how much autonomy they should be granted is biased and makes interpretation of this competency more challenging. [18][19][20] This PED CBE could be adapted to other clinical teaching experiences, with the caveat that there is likely variation between different specialties regarding teaching expectations. Our future goals are to assess reliability after a full year of implementation and investigate the role of CBEs in departmental educational offerings for faculty. The ultimate goal is improvement in faculty teaching behaviors, progressing to Kirkpatrick Level 3 (behavior) outcomes. 17 There are important considerations for other programs hoping to implement faculty CBEs. As the goal of the CBE is for individual skill development, faculty buy-in is critical to successful implementation. Faculty were briefed on the change prior to implementation and were educated on the role of CBEs in professional development. Notably, this assessment is from the perspective of trainees and must be paired with direct observation, and peer and supervisor evaluations to create a complete assessment of a teaching competency. Additionally, it is important to consider the limitations of the evaluation management system when developing a CBE, as the system needs to support behavioral anchor descriptions.
In summary, this was an impactful and feasible intervention of a faculty competency-based evaluation in our pediatric emergency department, including two new procedural sub-competencies, that was well received by trainees and faculty members. Conflicts of Interest: By the WestJEM article submission agreement, all authors are required to disclose all affiliations, funding sources and financial or management relationships that could be perceived as potential sources of bias. No author has professional or financial relationships with any companies that are relevant to this study. There are no conflicts of interest or sources of funding to declare.