Inter-rater reliability in toxicity identification: Limitations of current standards.
- Author(s): Fairchild, Andrew T
- Tanksley, Jarred P
- Tenenbaum, Jessica D
- Palta, Manisha
- Hong, Julian C
- et al.
Published Web Locationhttps://doi.org/10.1016/j.ijrobp.2020.04.040
PURPOSE:The NCI Common Terminology Criteria for Adverse Events (CTCAE) v5.0 is the standard for oncology toxicity encoding and grading despite limited validation. We assessed inter-rater reliability (IRR) in multi-reviewer toxicity identification. METHODS AND MATERIALS:Two reviewers independently reviewed 100 randomly selected notes for weekly on-treatment visits during radiotherapy from the electronic health record (EHR). Discrepancies were adjudicated by a third reviewer for consensus. Term harmonization was performed to account for overlapping symptoms in CTCAE. IRR was assessed based on unweighted and weighted Cohen's kappa coefficients. RESULTS:Between reviewers, unweighted kappa was 0.68 (95% CI 0.65-0.71) and weighted kappa 0.59 (0.22-1.00). IRR was consistent between noted present or absent symptoms with kappa of 0.6 (0.66-0.71) and 0.6 (0.65-0.69), respectively. CONCLUSIONS:Significant discordance suggests toxicity identification, particularly retrospectively, is a complex and error prone task. Strategies to minimize IRR, including training and simplification of the CTCAE criteria, should be considered in trial design and future terminologies.