This is the second in an article series contributed by Bülent Bayram, Tolga Bakirman, Esra Sunker, and Sevgi Zümra Karaca, Yildiz Technical University Department of Geomatic Engineering, Istanbul, Turkey; and, Buket Bayram, M.D., Dermatology Clinic, Istanbul, Turkey.
In a recent skin cancer research study, Dr. Bülent Bayram and his team at Yildiz Technical University, Department of Geomatics in Istanbul, Turkey, used Plainsight’s vision AI platform to perform image segmentation for early detection and analysis of skin cancer. As experts in photogrammetry, image processing, and machine learning, the Yildiz team is leveraging Plainsight vision AI to aid their work by facilitating efficient and accurate data curation for their research. Read about their continued advancement using our platform in this post.
_______
Part 2 in a blog series
MODEL TRAINING FOR MEDICAL RESEARCH WITH SMARTML
In the first part of this blog series, we created a Melanoma segmentation dataset with the images from the ISIC 2019 dataset using the Plainsight platform.
Considering the large number of images and the challenges of manual labeling, we have preferred the Plainsight platform and its features (such as SmartPoly and TrackForward) for accurately labeling each lesion’s boundary. In the end, a Melanoma segmentation dataset with 2793 images was created. Sample images from the dataset can be seen in the figure below.
Plainsight’s platform offers SmartML for users to train their models in the cloud. No computation power is needed on the client side and users don’t need to enter any code. We have evaluated the SmartML module with the new Melanoma segmentation dataset.
We have also created two subset datasets from our main Melanoma segmentation to assess the performance of models on datasets of different sizes. These are Dataset-1 and Dataset-2, consisting of 1730 and 1013 images, respectively. Once we are done with dataset configuration, we can now move on to training.
SmartML provides model outputs for classification, regression, bounding boxes, and polygons. Since we want to extract boundaries of Melanoma lesions, we chose the polygon option. Users have the ability to set the training time for models up to 24 hours. The choice will mainly be based on the size of the dataset and the backbone of the model. SmartML has an automatic early-stopping function to terminate the training once the model is converged to avoid overfitting. In our case, 2 hours were sufficient for our models to train. If you do not have a background in deep learning applications, you can go on and start your training without additional adjustments. In this example, we want to set our hyperparameters for our network and conduct various experiments to assess how these parameters affect the model results.
The first parameter is to decide how to split the dataset into training, validation, and test sets. By default, these ratios are 80%, 10%, and 10%, respectively. These can be optionally set as 70%, 20%, and 10%. SmartML offers three feature pyramid network (FPN)-based backbones: R50 FPN, R101 FPN, and X101 FPN. While the X101 backbone is based on the ResNeXt model with 101 layers, the R50 and R101 backbones are based on ResNet models with 50 and 101 layers, respectively. In this example, R50 and R101 backbones are chosen to evaluate how the number of layers affects the model’s performance for Melanoma segmentation. Learning rate and batch size can also be set as they are essential hyperparameters that affect the training process. We have also used horizontal flip and shift-scale-rotate augmentation techniques to increase our dataset size. In total, we have conducted 6 experiments with Dataset-1 and Dataset-2. Hyperparameters and results can be seen in the table below.
The best accuracy and loss metrics were obtained with Dataset-1 trained with R50 FPN and a batch size of 20. The results of other experiments conducted with Dataset-1 are quite similar. However, the number of images in Dataset-2 does not seem to be sufficient considering the accuracy metrics for all three experiments. SmartML shows the model’s training graph for better evaluation of the results. The figure below shows the training graph for our best model.
We can also run inference to make predictions on test images. Here are some prediction examples from the test dataset of Dataset-1.
The figure below shows predictions from the Dataset-2. You can see that the performance is not sufficient at all. This shows that the number of images in the dataset is crucial for effective model training.
Plainsight’s SmartML function was found to be very useful by offering a no-code method to train these critical prediction models in the cloud within just a few hours. The ease of use and speed in training these models can improve skin cancer diagnosis, which could ultimately help prevent mortality. This study and partnership aims to solve these challenges and to provide a second opinion to physicians, increasing the accuracy and ultimately speed of their diagnoses.
