Skip to main content
eScholarship
Open Access Publications from the University of California

UCSF

UC San Francisco Previously Published Works bannerUCSF

Interrater Reliability in Toxicity Identification: Limitations of Current Standards.

  • Author(s): Fairchild, Andrew T
  • Tanksley, Jarred P
  • Tenenbaum, Jessica D
  • Palta, Manisha
  • Hong, Julian C
  • et al.
Abstract

Purpose

The National Cancer Institute Common Terminology Criteria for Adverse Events (CTCAE) v5.0 is the standard for oncology toxicity encoding and grading, despite limited validation. We assessed interrater reliability (IRR) in multireviewer toxicity identification.

Methods and materials

Two reviewers independently reviewed 100 randomly selected notes for weekly on-treatment visits during radiation therapy from the electronic health record. Discrepancies were adjudicated by a third reviewer for consensus. Term harmonization was performed to account for overlapping symptoms in CTCAE. IRR was assessed based on unweighted and weighted Cohen's kappa coefficients.

Results

Between reviewers, the unweighted kappa was 0.68 (95% confidence interval, 0.65-0.71) and the weighted kappa was 0.59 (0.22-1.00). IRR was consistent between symptoms noted as present or absent with a kappa of 0.6 (0.66-0.71) and 0.6 (0.65-0.69), respectively.

Conclusions

Significant discordance suggests toxicity identification, particularly retrospectively, is a complex and error-prone task. Strategies to minimize IRR, including training and simplification of the CTCAE criteria, should be considered in trial design and future terminologies.

Many UC-authored scholarly publications are freely available on this site because of the UC's open access policies. Let us know how this access is important for you.

Main Content
Current View