Concordance in Basal Cell Carcinoma Diagnosis. Building a Proper Standard Reference to Train Artificial Intelligence Tools.
Reliable labels are essential when training Artificial Intelligence (AI) tools. Whereas some diseases allow biopsy-based labeling, others rely on subjective criteria. For the diagnosis of basal cell carcinoma (BCC), dermatologists detect certain dermoscopic criteria, whose presence (or absence) serves as the basis for determining a diagnosis of BCC. Therefore, an AI tool assisting in BCC diagnosis should provide such criteria to explain its output.
This study analyzes the agreement among four dermatologists in detecting dermoscopic criteria and compares the performance of an AI model trained with labels from a single dermatologist versus a consensus-based standard. A total of1230 dermoscopic images, collected in around 60 primary health centers, sent via teledermatology, and diagnosed by four dermatologists, were used to train an AI tool. They were randomly selected from the teledermatology platform (2019-2021). Subsequently, 204 new images were used to test the AI tool prospectively. A standard reference (SR) was built using Expectation Maximization on the four diagnoses. The performance of the AI tool trained using the reference standard of one dermatologist versus the reference standard statistically inferred from the consensus of four dermatologists was analyzed using McNemar's test and Hamming distance.
Agreement among dermatologists was high for BCC versus non-BCC (Kappa = 0.9079; PPV = 0.9670), but lower for specific criteria. Statistical differences were found in the performance of AI models trained with individual and consensus labels.
Deriving an SR from multiple expert opinions mitigates individual bias and enhances AI interpretability, key for its clinical adoption.
This study analyzes the agreement among four dermatologists in detecting dermoscopic criteria and compares the performance of an AI model trained with labels from a single dermatologist versus a consensus-based standard. A total of1230 dermoscopic images, collected in around 60 primary health centers, sent via teledermatology, and diagnosed by four dermatologists, were used to train an AI tool. They were randomly selected from the teledermatology platform (2019-2021). Subsequently, 204 new images were used to test the AI tool prospectively. A standard reference (SR) was built using Expectation Maximization on the four diagnoses. The performance of the AI tool trained using the reference standard of one dermatologist versus the reference standard statistically inferred from the consensus of four dermatologists was analyzed using McNemar's test and Hamming distance.
Agreement among dermatologists was high for BCC versus non-BCC (Kappa = 0.9079; PPV = 0.9670), but lower for specific criteria. Statistical differences were found in the performance of AI models trained with individual and consensus labels.
Deriving an SR from multiple expert opinions mitigates individual bias and enhances AI interpretability, key for its clinical adoption.
Authors
Silva-Clavería Silva-Clavería, Serrano Serrano, Matas Matas, Serrano Serrano, Toledo-Pastrana Toledo-Pastrana, Acha Acha
View on Pubmed