Skip to main content
Open Access Publications from the University of California

Inter-rater reliability in toxicity identification: Limitations of current standards.

  • Author(s): Fairchild, Andrew T
  • Tanksley, Jarred P
  • Tenenbaum, Jessica D
  • Palta, Manisha
  • Hong, Julian C
  • et al.

PURPOSE:The NCI Common Terminology Criteria for Adverse Events (CTCAE) v5.0 is the standard for oncology toxicity encoding and grading despite limited validation. We assessed inter-rater reliability (IRR) in multi-reviewer toxicity identification. METHODS AND MATERIALS:Two reviewers independently reviewed 100 randomly selected notes for weekly on-treatment visits during radiotherapy from the electronic health record (EHR). Discrepancies were adjudicated by a third reviewer for consensus. Term harmonization was performed to account for overlapping symptoms in CTCAE. IRR was assessed based on unweighted and weighted Cohen's kappa coefficients. RESULTS:Between reviewers, unweighted kappa was 0.68 (95% CI 0.65-0.71) and weighted kappa 0.59 (0.22-1.00). IRR was consistent between noted present or absent symptoms with kappa of 0.6 (0.66-0.71) and 0.6 (0.65-0.69), respectively. CONCLUSIONS:Significant discordance suggests toxicity identification, particularly retrospectively, is a complex and error prone task. Strategies to minimize IRR, including training and simplification of the CTCAE criteria, should be considered in trial design and future terminologies.

Many UC-authored scholarly publications are freely available on this site because of the UC's open access policies. Let us know how this access is important for you.

Main Content
Current View