Validation of a Text-Mining Tool for Extracting Routine Clinical Care Data in Early-Stage Resectable Non-Small Cell Lung Cancer.
Manual chart review (MR) of electronic health records (EHRs) is time-consuming, error-prone, and limits the reproducibility and scalability of real-world data (RWD) research. Automation and standardization using natural language processing (NLP) could improve efficiency and scalability. CTcue is an NLP-based software platform designed to extract structured and unstructured data from EHRs. This study evaluated the accuracy and efficiency of CTcue versus MR in patients with early-stage resectable non-small cell lung cancer (NSCLC).
Included were all patients with stage I to III NSCLC who underwent lung resections between January 2018 and December 2021 at the Leiden University Medical Center, the Netherlands. Demographics, tumor characteristics, treatment, and outcomes were collected. CTcue performance was compared with MR using weighted F1-scores, accuracy, precision, and recall for categorical variables and Bland-Altman analysis for continuous variables.
Eighty-five patients (70.2% of patients from the manual cohort) were identified by both methods and included in the comparison. CTcue achieved weighted F1-scores >0.85 for seven of 15 categorical variables, including sex, tumor location, and deceased status, although some scores were based on low number of observations in both cohorts. Lower performance was observed for variables with varying terminology in documentation, such as Eastern Cooperative Oncology Group status and pathological N-stage. Continuous variables showed negligible mean differences, indicating good agreement. Survival outcomes were identical in both data sets.
CTcue performance for patient selection was lower than anticipated. However, it enables accurate, efficient extraction of structured and unstructured EHR data in early-stage NSCLC. Manual validation remains necessary for variables with varying terminology. Further development of artificial intelligence-based tools-particularly for free-text data extraction-will be crucial to enhance the accuracy and scalability of future RWD research.
Included were all patients with stage I to III NSCLC who underwent lung resections between January 2018 and December 2021 at the Leiden University Medical Center, the Netherlands. Demographics, tumor characteristics, treatment, and outcomes were collected. CTcue performance was compared with MR using weighted F1-scores, accuracy, precision, and recall for categorical variables and Bland-Altman analysis for continuous variables.
Eighty-five patients (70.2% of patients from the manual cohort) were identified by both methods and included in the comparison. CTcue achieved weighted F1-scores >0.85 for seven of 15 categorical variables, including sex, tumor location, and deceased status, although some scores were based on low number of observations in both cohorts. Lower performance was observed for variables with varying terminology in documentation, such as Eastern Cooperative Oncology Group status and pathological N-stage. Continuous variables showed negligible mean differences, indicating good agreement. Survival outcomes were identical in both data sets.
CTcue performance for patient selection was lower than anticipated. However, it enables accurate, efficient extraction of structured and unstructured EHR data in early-stage NSCLC. Manual validation remains necessary for variables with varying terminology. Further development of artificial intelligence-based tools-particularly for free-text data extraction-will be crucial to enhance the accuracy and scalability of future RWD research.
Authors
Abedian Kalkhoran Abedian Kalkhoran, Martinot Martinot, Schonewille Schonewille, Huyuk Huyuk, Huigen Huigen, Guchelaar Guchelaar, Cohen Cohen, Smit Smit, Zwaveling Zwaveling
View on Pubmed