Validation of a performance checklist for ultrasound-guided internal jugular central lines for use in procedural instruction and assessment

Purpose of the study Tools created to measure procedural competency must be tested in their intended environment against an established standard in order to be validated. We previously created a checklist for ultrasound-guided internal jugular central venous catheter (US IJ CVC) insertion using the modified Delphi method. We sought to further validate the checklist tool for use in an educational environment. Study design This is a cohort study involving 15 emergency medicine interns being evaluated on their skill in US IJ CVC placement. We compared the checklist tool with a modified version of a clinically validated global rating scale (GRS) for procedural performance. Results The correlation between the GRS tool and the checklist tool was excellent, with a correlation coefficient (Pearson's r) of 0.90 (p<0.0001). Conclusions This checklist represents a useful tool for measuring procedural competency.


BACKGROUND
Effective assessment tools are essential to residency education. In order to provide feedback and teaching for resident learners, faculty members must be able to assess resident performance of critical procedures in a standardised fashion. The performance of critical procedures, including central line insertion, is one domain in which effective instruction and adherence to guideline specifications can lead directly to improved patient outcomes and enhanced quality of care. 1 2 Recent research in graduate medical education has confirmed that competency-centred instruction and assessment using validated assessment tools can lead to improved quality of care, including improved performance and fewer complications from central line insertion. 3 We have created and validated a checklist tool for performance of ultrasound-guided internal jugular central venous catheter (US IJ CVC) placement using a modified Delphi method. 4 The Delphi method is a procedure in which expert opinion is gathered to reach consensus in a structured fashion. In order to test the checklist's applicability to the educational realm, it should be tested in actual use for that purpose.
We sought to validate the US IJ CVC checklist tool in order to evaluate resident competency in procedure performance. To this end, we analysed data collected during a study on effective feedback strategies for resident learners performing US IJ CVC placement. Residents were assessed using both the US IJ CVC checklist tool and a previously validated global rating scale (GRS) used in the surgical literature for procedural performance evaluation. 5 6 To the study team's knowledge, no validation of this US IJ CVC checklist tool has been completed in an educational setting. By comparing individual scores on these instruments, we assessed the validity of the checklist for educational use in procedural instruction.

Purpose of the study
To evaluate a checklist tool for assessment of resident skill in US IJ CVC placement. We hypothesise that a checklist score ascertained for resident performance of this skill will highly correlate with a modified version of a previously validated GRS for procedural performance.

Study design
We compared the rating scores between the US IJ CVC checklist tool and a modified version of a previously validated GRS for procedural performance. The GRS is used in the surgical services as a validated tool to evaluate procedural competency (appendix A). This tool was minimally modified by selecting those sections that applied to a bedside procedure, resulting in six criteria with a maximum of five points each. The first item removed was 'respect for tissue', secondary to lack of invasive operative technique and use of a synthetic model. The second item removed was 'use of assistants' as the studied intervention did not require an assistant. The checklist is a step by step outline of the procedure (appendix B), with each checked item counting as a point. The items covered do not overlap between the evaluation tools. The purpose of the study was to quantify the degree of correlation between the checklist tool and the GRS.

Study protocol
We completed a prospective study involving procedural skill evaluation and feedback on resident performance of US IJ CVC in a simulated environment, involving 15 emergency medicine interns at an academic medical centre in July-August 2014. During the study, each intern performed US IJ CVC placement twice, with two faculty instructors evaluating procedural skill and providing feedback. The intern initially performed the US IJ CVC placement while observed by the two clinical faculty members. The faculty and the intern then independently completed a GRS for the procedure, resulting in three GRS scores. The faculty subsequently jointly evaluated and provided feedback using the US IJ CVC checklist tool. 4 After receiving feedback and having time to practice, each intern repeated the procedure. The participant did not have access to the checklist or other notes during either attempt. The scoring process was repeated, resulting in three GRS scores and a checklist tool score for the post-phase. Participants were also surveyed to assess their previous experience with CVC insertion.

Data analysis
Agreement between faculty raters for the GRS was assessed using a quadratic weighted kappa using bias-corrected bootstrap CIs. This method is appropriate for ordinal categorical data with an anticipated high number of tied assessments and increases the weight of the relative disagreement between the assessors with increasing degrees of disagreement. As the distribution of the sum of the GRS element scores and the checklist tool satisfied a normal distribution, as assessed by graphic inspection of the data as well as the Shapiro-Wilk test, Pearson's r was used to assess the correlation between the average of the faculty score on the GRS and the corresponding checklist score for each episode. A two-tailed paired t-test was used to compare checklist scores before and after feedback. A p value of 0.05 was held to be statistically significant. Statistics were calculated using Stata IC V.11.2.

RESULTS
Each resident performed two US IJ CVC placements for 30 total procedures. The mean initial score on the checklist tool was 16.9, SD 4.7. The mean checklist score after feedback was 26.5, SD 2.0. All residents improved scores on the checklist tool with feedback, with a mean increase of 9.6 (range 5-15, 95% CI 7.8 to 11.4, p<0.0001). Prior to this training experience, participants had completed a mean of 1.6 simulated CVC insertions and 1.1 live CVC insertions.
The quadratic weighted kappa for the faculty GRS scores was excellent, with kappa of 0.79 (95% CI 0.75 to 0.84). The correlation between the GRS scores and the checklist scores was very high, with a Pearson's r=0.90 ( p<0.0001) for the first CVC placement and 0.89 ( p<0.0001) for the second CVC placement.

CONCLUSIONS
The US IJ CVC checklist tool for resident performance was highly correlated with a validated GRS, which itself demonstrated excellent inter-rater reliability, consistent with prior performance for that tool. A previous study using the GRS instrument showed a reliability coefficient of 0.78, suggesting consistent inter-rater reliability. 5 7 This investigation supports several strengths of this tool for use in educational settings. Despite the significant change in the interns' average rating between the first and second attempt, which likely stems partially from the participants' prior relative inexperience, the two measures continued to be highly correlated. This suggests that the correlation noted here does not depend strongly on the skill level of the trainee.
Anecdotally, the checklist also functioned well as a tool for instruction and feedback as it allowed faculty instructors to systematically review the procedure while addressing all critical steps and any noted areas of deficiency. Using two different assessment tools, one with dichotomous items marked for completion, the other with ordinal ratings of technical skill, also allowed instructors to provide both objective and subjective feedback and assessment. The correlation between the two also suggests that either can be used and represent a valid assessment of the resident's skill and progress. This approach, combining a GRS with a procedural component checklist tool, has been well established as a pedagogical and evaluation tool in graduate medical education, particularly with respect to procedures. 8 Using measures that are validated provides needed strength to these educational endeavours.
Validated and practical tools for assessment are vital to the improvement of resident education and quality of care. In the absence of these tools, educators are left with impression and possibly faulty measurements of resident skills. The graduate medical education literature is clear that resident self-assessment is often flawed and depends highly on the learner's skill level. 9 10 By creating and scrutinising assessment tools, the consistency and quality of medical encounters and procedures can be improved.

LIMITATIONS
As the same procedure was repeated after feedback using the procedural checklist, the bias of repeated measures may have influenced the rating. However, this is true for both the GRS and the checklist that strongly correlated in both the first and second CVC insertion for each resident, during which the residents received significantly different scores. The GRS used here is slightly modified from the previously validated version, done so as to more accurately apply to a non-surgical procedure. The changes do not seem significant enough to cause disruption, and the inter-rater performance of the tool is consistent with prior reports.
The faculty performed their own checklist rating on the intern, but the two independent evaluations were combined into one single rating so that appropriate feedback could be delivered. This did not allow for us to check the inter-rater

Main messages
▸ Insertion of central venous catheters in the internal jugular vein has been made significantly safer with the advent of ultrasound (US) guidance. ▸ A validated checklist for the insertion of central lines in the internal jugular using US has been created using the modified Delphi method. ▸ Training programmes are increasingly asked to attest to trainee competence in critical skills and require validated tools to do so. ▸ The checklist tool for this procedure performed very well when compared against an already validated tool for procedural assessment and appears to be a useful, validated tool to assess trainee skill in a critical procedure.

Current research questions
▸ Does use of this instrument as a training tool lead to improved skill in central line placement? ▸ Can this tool be validated against other tools used for procedural assessment? ▸ Does use of this checklist lead to improved patient outcomes in the clinical setting?
reliability between the two raters for the procedural checklist evaluation tool.