Can physics review catch prescription and PTV margin errors in fully automated radiotherapy plans? A hazard study.
End-to-end AI-based automated contouring and radiotherapy planning systems promise substantial gains in efficiency and consistency but also introduce new safety risks, including user input errors that occur early in the workflow. This study evaluates the ability of standard physics plan review to detect user errors in prescription and planning target volume (PTV) margin entry in a fully automated planning workflow.
A prototype of the Radiation Planning Assistant (RPA) was used to generate fully automated prostate cancer treatment plans for 20 patients. Five cases intentionally included incorrect prescriptions or non-standard PTV margins. Five clinical medical physicists independently performed physics plan reviews in a commercial treatment planning system, assuming a standard clinical workflow and without access to user guides or service request forms. Reviews were conducted in two phases, with and without margin information embedded in structure names. A third phase evaluated whether adding reference expansion structures improved margin error detection.
Incorrect prescriptions were detected in 80% of cases, with no false-positive identification of correct prescriptions. In contrast, detection of incorrect PTV margins was highly inconsistent. Two physicists did not evaluate margins and missed all margin errors. Three physicists identified all intentional margin errors but demonstrated poor specificity, frequently flagging correct cases as incorrect (specificity range: 13%-63%). Including margin information in structure names and adding reference expansion structures did not improve specificity or overall detection performance.
Physics plan review was unreliable for detecting incorrect PTV margins in a fully automated planning workflow, even when additional cues were provided. Although based on a limited dataset, these findings indicate that traditional plan review processes alone are insufficient to mitigate this failure mode. Alternative risk-reduction strategies-such as timely workflow-embedded alerts, targeted checklists, automated verification tools, and careful expectation setting-are likely required to improve safety in AI-driven radiotherapy planning systems.
A prototype of the Radiation Planning Assistant (RPA) was used to generate fully automated prostate cancer treatment plans for 20 patients. Five cases intentionally included incorrect prescriptions or non-standard PTV margins. Five clinical medical physicists independently performed physics plan reviews in a commercial treatment planning system, assuming a standard clinical workflow and without access to user guides or service request forms. Reviews were conducted in two phases, with and without margin information embedded in structure names. A third phase evaluated whether adding reference expansion structures improved margin error detection.
Incorrect prescriptions were detected in 80% of cases, with no false-positive identification of correct prescriptions. In contrast, detection of incorrect PTV margins was highly inconsistent. Two physicists did not evaluate margins and missed all margin errors. Three physicists identified all intentional margin errors but demonstrated poor specificity, frequently flagging correct cases as incorrect (specificity range: 13%-63%). Including margin information in structure names and adding reference expansion structures did not improve specificity or overall detection performance.
Physics plan review was unreliable for detecting incorrect PTV margins in a fully automated planning workflow, even when additional cues were provided. Although based on a limited dataset, these findings indicate that traditional plan review processes alone are insufficient to mitigate this failure mode. Alternative risk-reduction strategies-such as timely workflow-embedded alerts, targeted checklists, automated verification tools, and careful expectation setting-are likely required to improve safety in AI-driven radiotherapy planning systems.
Authors
Court Court, Douglas Douglas, Diagaradjane Diagaradjane, Kathriarachchi Kathriarachchi, S S, Subashi Subashi, Zhao Zhao, Zhang Zhang, Netherton Netherton
View on Pubmed