Reproducibility in deep learning algorithms for digital pathology applications: a case study using the CAMELYON16 datasets

Weizhe Li; Weijie Chen

doi:10.1117/12.2581996

15 February 2021 Reproducibility in deep learning algorithms for digital pathology applications: a case study using the CAMELYON16 datasets

Weizhe Li, Weijie Chen

Author Affiliations +

Proceedings Volume 11603, Medical Imaging 2021: Digital Pathology; 1160318 (2021) https://doi.org/10.1117/12.2581996
Event: SPIE Medical Imaging, 2021, Online Only

Conference Poster

Abstract

Reproducibility as a cornerstone of science has in recent years attracted significant attention in a variety of applications that involve big data and complex algorithms. Artificial Intelligence (AI) applications, in particular, have been referred by some authors as having a “reproducibility crisis”. The application of deep learning algorithms in digital pathology, which has been demonstrated in some studies to achieve pathologist-level performance, involves both big data and complex algorithms. It is important to identify technical factors influencing AI performance for such studies to be reproducible. In this work, we conducted a reproducibility study using the public datasets shared by the CAMELYON16 challenge, which aimed to develop and assess algorithms for the detection of breast cancer metastasis using whole-slide images (WSIs) of lymph node sections. We used the two-stage classification framework of the CAMELYON16 challenge’s top-performing algorithm, i.e., a convolutional neural network (Inception V1) to generate heatmaps followed by feature extraction from the heatmaps and a random forest classifier for classification. We investigated the effects of variations in training/testing procedures on the performance of AI/ML algorithms including color augmentation methods, color normalization methods, and the resolution of heatmaps. Our results showed that, despite the differences in color augmentation and color normalization methods, the utilization of these techniques improved the classification performance by an AUC value of 0.07 compared to without using them, which is consistent with the CAMELYON16 findings. We concluded that sufficient details of technical description should be provided for a study to be fully reproducible.

Conference Presentation

Citation Download Citation

Weizhe Li and Weijie Chen "Reproducibility in deep learning algorithms for digital pathology applications: a case study using the CAMELYON16 datasets", Proc. SPIE 11603, Medical Imaging 2021: Digital Pathology, 1160318 (15 February 2021); https://doi.org/10.1117/12.2581996

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available