PURPOSE: Automation and computer assistance can support quality assurance tasks in radiotherapy. Retrospective image review requires significant human resources, and automation of image review remains a noteworthy missing element in previous work. Here, we present initial findings from a proof-of-concept clinical implementation of an AI-assisted review of CBCT registrations used for patient setup. METHODS: An automated pipeline was developed and executed nightly, utilizing python scripts to interact with the clinical database through DICOM networking protocol and automate data retrieval and analysis. A previously developed artificial intelligence (AI) algorithm scored CBCT setup registrations based on misalignment likelihood, using a scale from 0 (most unlikely) through 1 (most likely). Over a 45-day period, 1357 pre-treatment CBCT registrations from 197 patients were retrieved and analyzed by the pipeline. Daily summary reports of the previous days registrations were produced. Initial action levels targeted 10% of cases to highlight for in-depth physics review. A validation subset of 100 cases was scored by three independent observers to characterize AI-model performance. RESULTS: Following an ROC analysis, a global threshold for model predictions of 0.87 was determined, with a sensitivity of 100% and specificity of 82%. Inspecting the observer scores for the stratified validation dataset showed a statistically significant correlation between observer scores and model predictions. CONCLUSION: In this work, we describe the implementation of an automated AI-analysis pipeline for daily quantitative analysis of CBCT-guided patient setup registrations. The AI-model was validated against independent expert observers, and appropriate action levels were determined to minimize false positives without sacrificing sensitivity. Case studies demonstrate the potential benefits of such a pipeline to bolster quality and safety programs in radiotherapy. To the authors knowledge, there are no previous works performing AI-assisted assessment of pre-treatment CBCT-based patient alignment.

Background

This study utilizes interviews of clinical medical physicists to investigate self-reported shortcomings of the current weekly chart check workflow and opportunities for improvement.

Methods

Nineteen medical physicists were recruited for a 30-minute semi-structured interview, with a particular focus placed on image review and the use of automated tools for image review in weekly checks. Survey-type questions were used to gather quantitative information about chart check practices and importance placed on reducing chart check workloads versus increasing chart check effectiveness. Open-ended questions were used to probe respondents about their current weekly chart check workflow, opinions of the value of weekly chart checks and perceived shortcomings, and barriers and facilitators to the implementation of automated chart check tools. Thematic analysis was used to develop common themes across the interviews.

Results

Physicists ranked highly the value of reducing the time spent on weekly chart checks (average 6.3 on a scale from 1 to 10), but placed more value on increasing the effectiveness of checks with an average of 9.2 on a 1-10 scale. Four major themes were identified: (1) weekly chart checks need to adapt to an electronic record-and-verify chart environment, (2) physicists could add value to patient care by analyzing images without duplicating the work done by physicians, (3) greater support for trending analysis is needed in weekly checks, and (4) automation has the potential to increase the value of physics checks.

Conclusion

This study identified several key shortcomings of the current weekly chart check process from the perspective of the clinical medical physicist. Our results show strong support for automating components of the weekly check workflow in order to allow for more effective checks that emphasize follow-up, trending, failure modes and effects analysis, and allow time to be spent on other higher value tasks that improve patient safety.

Purpose

Present knowledge of patient setup and alignment errors in image guided radiation therapy (IGRT) relies on voluntary reporting, which is thought to underestimate error frequencies. A manual retrospective patient-setup misalignment error search is infeasible owing to the bulk of cases to be reviewed. We applied a deep learning-based misalignment error detection algorithm (EDA) to perform a fully automated retrospective error search of clinical IGRT databases and determine an absolute gross patient misalignment error rate.

Methods and materials

The EDA was developed to analyze the registration between planning scans and pretreatment cone beam computed tomography scans, outputting a misalignment score ranging from 0 (most unlikely) to 1 (most likely). The algorithm was trained using simulated translational errors on a data set obtained from 680 patients treated at 2 radiation therapy clinics between 2017 and 2022. A receiver operating characteristic analysis was performed to obtain target thresholds. DICOM Query and Retrieval software was integrated with the EDA to interact with the clinical database and fully automate data retrieval and analysis during a retrospective error search from 2016 to 2017 and from 2021 to 2022 for the 2 institutions, respectively. Registrations were flagged for human review using both a hard-thresholding method and a prediction trending analysis over each individual patient's treatment course. Flagged registrations were manually reviewed and categorized as errors (>1 cm misalignment at the target) or nonerrors.

Results

A total of 17,612 registrations were analyzed by the EDA, resulting in 7.7% flagged events. Three previously reported errors were successfully flagged by the EDA, and 4 previously unreported vertebral body misalignment errors were discovered during case reviews. False positive cases often displayed substantial image artifacts, patient rotation, and soft tissue anatomy changes.

Conclusions

Our results validated the clinical utility of the EDA for bulk image reviews and highlighted the reliability and safety of IGRT, with an absolute gross patient misalignment error rate of 0.04% ± 0.02% per delivered fraction.

Background

In cone-beam computed tomography (CBCT)-guided radiotherapy, off-by-one vertebral-body misalignments are rare but serious errors that lead to wrong-site treatments.

Purpose

An automatic error detection algorithm was developed that uses a three-branch convolutional neural network error detection model (EDM) to detect off-by-one vertebral-body misalignments using planning computed tomography (CT) images and setup CBCT images.

Methods

Algorithm training and test data consisted of planning CTs and CBCTs from 480 patients undergoing radiotherapy treatment in the thoracic and abdominal regions at two radiotherapy clinics. The clinically applied registration was used to derive true-negative (no error) data. The setup and planning images were then misaligned by one vertebral-body in both the superior and inferior directions, simulating the most likely misalignment scenarios. For each of the aligned and misaligned 3D image pairs, 2D slice pairs were automatically extracted in each anatomical plane about a point within the vertebral column. The three slice pairs obtained were then inputted to the EDM that returned a probability of vertebral misalignment. One model (EDM₁ ) was trained solely on data from institution 1. EDM₁ was further trained using a lower learning rate on a dataset from institution 2 to produce a fine-tuned model, EDM₂ . Another model, EDM₃ , was trained from scratch using a training dataset composed of data from both institutions. These three models were validated on a randomly selected and unseen dataset composed of images from both institutions, for a total of 303 image pairs. The model performances were quantified using a receiver operating characteristic analysis. Due to the rarity of vertebral-body misalignments in the clinic, a minimum threshold value yielding a specificity of at least 99% was selected. Using this threshold, the sensitivity was calculated for each model, on each institution's test set separately.

Results

When applied to the combined test set, EDM₁ , EDM₂ , and EDM₃ resulted in an area under curve of 99.5%, 99.4%, and 99.5%, respectively. EDM₁ achieved a sensitivity of 96% and 88% on Institution 1 and Institution 2 test set, respectively. EDM₂ obtained a sensitivity of 95% on each institution's test set. EDM₃ achieved a sensitivity of 95% and 88% on Institution 1 and Institution 2 test set, respectively.

Conclusion

The proposed algorithm demonstrated accuracy in identifying off-by-one vertebral-body misalignments in CBCT-guided radiotherapy that was sufficiently high to allow for practical implementation. It was found that fine-tuning the model on a multi-facility dataset can further enhance the generalizability of the algorithm.

PURPOSE: To quantify variations in target and normal structure contouring and evaluate dosimetric impact of these variations in non-small cell lung cancer (NSCLC) cases. To study whether providing an atlas can reduce potential variation. METHODS AND MATERIALS: Three NSCLC cases were distributed sequentially to multiple institutions for contouring and radiation therapy planning. No segmentation atlas was provided for the first 2 cases (Case 1 and Case 2). Contours were collected from submitted plans and consensus contour sets were generated. The volume variation among institution contours and the deviation of them from consensus contours were analyzed. The dose-volume histograms for individual institution plans were recalculated using consensus contours to quantify the dosimetric changes. An atlas containing targets and critical structures was constructed and was made available when the third case (Case 3) was distributed for planning. The contouring variability in the submitted plans of Case 3 was compared with that in first 2 cases. RESULTS: Planning target volume (PTV) showed large variation among institutions. The PTV coverage in institutions plans decreased dramatically when reevaluated using the consensus PTV contour. The PTV contouring consistency did not show improvement with atlas use in Case 3. For normal structures, lung contours presented very good agreement, while the brachial plexus showed the largest variation. The consistency of esophagus and heart contouring improved significantly (t test; P < .05) in Case 3. Major factors contributing to the contouring variation were identified through a survey questionnaire. CONCLUSIONS: The amount of contouring variations in NSCLC cases was presented. Its impact on dosimetric parameters can be significant. The segmentation atlas improved the contour agreement for esophagus and heart, but not for the PTV in this study. Quality assurance of contouring is essential for a successful multi-institutional clinical trial.

Purpose

To quantify variations in target and normal structure contouring and evaluate dosimetric impact of these variations in non-small cell lung cancer (NSCLC) cases. To study whether providing an atlas can reduce potential variation.

Methods and materials

Three NSCLC cases were distributed sequentially to multiple institutions for contouring and radiation therapy planning. No segmentation atlas was provided for the first 2 cases (Case 1 and Case 2). Contours were collected from submitted plans and consensus contour sets were generated. The volume variation among institution contours and the deviation of them from consensus contours were analyzed. The dose-volume histograms for individual institution plans were recalculated using consensus contours to quantify the dosimetric changes. An atlas containing targets and critical structures was constructed and was made available when the third case (Case 3) was distributed for planning. The contouring variability in the submitted plans of Case 3 was compared with that in first 2 cases.

Results

Planning target volume (PTV) showed large variation among institutions. The PTV coverage in institutions' plans decreased dramatically when reevaluated using the consensus PTV contour. The PTV contouring consistency did not show improvement with atlas use in Case 3. For normal structures, lung contours presented very good agreement, while the brachial plexus showed the largest variation. The consistency of esophagus and heart contouring improved significantly (t test; P < .05) in Case 3. Major factors contributing to the contouring variation were identified through a survey questionnaire.

Conclusions

The amount of contouring variations in NSCLC cases was presented. Its impact on dosimetric parameters can be significant. The segmentation atlas improved the contour agreement for esophagus and heart, but not for the PTV in this study. Quality assurance of contouring is essential for a successful multi-institutional clinical trial.

A substantial barrier to the single- and multi-institutional aggregation of data to supporting clinical trials, practice quality improvement efforts, and development of big data analytics resource systems is the lack of standardized nomenclatures for expressing dosimetric data. To address this issue, the American Association of Physicists in Medicine (AAPM) Task Group 263 was charged with providing nomenclature guidelines and values in radiation oncology for use in clinical trials, data-pooling initiatives, population-based studies, and routine clinical care by standardizing: (1) structure names across image processing and treatment planning system platforms; (2) nomenclature for dosimetric data (eg, dose-volume histogram [DVH]-based metrics); (3) templates for clinical trial groups and users of an initial subset of software platforms to facilitate adoption of the standards; (4) formalism for nomenclature schema, which can accommodate the addition of other structures defined in the future. A multisociety, multidisciplinary, multinational group of 57 members representing stake holders ranging from large academic centers to community clinics and vendors was assembled, including physicists, physicians, dosimetrists, and vendors. The stakeholder groups represented in the membership included the AAPM, American Society for Radiation Oncology (ASTRO), NRG Oncology, European Society for Radiation Oncology (ESTRO), Radiation Therapy Oncology Group (RTOG), Children's Oncology Group (COG), Integrating Healthcare Enterprise in Radiation Oncology (IHE-RO), and Digital Imaging and Communications in Medicine working group (DICOM WG); A nomenclature system for target and organ at risk volumes and DVH nomenclature was developed and piloted to demonstrate viability across a range of clinics and within the framework of clinical trials. The final report was approved by AAPM in October 2017. The approval process included review by 8 AAPM committees, with additional review by ASTRO, European Society for Radiation Oncology (ESTRO), and American Association of Medical Dosimetrists (AAMD). This Executive Summary of the report highlights the key recommendations for clinical practice, research, and trials.