Background

Publicly available skin image datasets are increasingly used to train and validate machine learning (ML) algorithms for skin cancer diagnosis. The concept of health data poverty and systematic data disparities leading to inequalities have recently been highlighted, emphasising the need to ensure transparency and usability of datasets. In addition to the images themselves, acquisition setting and associated metadata significantly influence ML algorithm development and validation for skin cancer diagnoses. Indeed, ML classification accuracy improves when clinical metadata are integrated. To evaluate the characteristics of publicly available datasets, we systematically reviewed those used to test and train ML algorithms for skin cancer diagnoses.

Method

Open-access skin lesion image datasets were identified using MEDLINE, Google and Google Dataset searches, with supplementary manual screening. Two independent reviewers performed searches (4/9/2020), and extracted characteristics and metadata for included datasets.

Results

Overall, 21 open-access datasets containing 106,950 skin lesion images were identified. Thirteen contained dermoscopic images, six contained macroscopic photographs, and two contained paired dermoscopic and macroscopic images (reflecting clinical practice). Fourteen datasets reported country of origin; nine contained images from Europe. Subject number was reported in 38.1%, data collection period in 38.1%, ethical approval in 47.6%, and participant consent in 33.3%.

Reviewing individual image metadata, clinical information were available for age (76.4%), gender (77.5%) and body site (74.4%). Ethnicity and Fitzpatrick skin type metadata were only available for 1.3% and 2.1% of images. Of 2,236 images with individual skin type labels, only eleven (0.5%) were darker-skinned individuals (type V-VI). Histopathology ground truth were available for 68.8% malignant lesions and 20.4% of all images.

Conclusion

This is the first systematic review of open-access skin image datasets, highlighting limited and variable metadata reporting, limited applicability of datasets to real-life clinical settings and restricted representation of populations, with implications for generalisability. This highlights the need for quality standards for characteristics and metadata reporting for skin image datasets.

Impact statement

There is an urgent need for publicly available image datasets to be transparent, have a minimum metadata requirement, and to represent the intended populations that they will be deployed in, in order to maximise accuracy and applicability of skin cancer detection algorithms.