Melanoma Detection through Combining Reinforcement Learning, Generative Adversarial Network, and Bayesian Optimization
Document Type
Article
Publication Date
2-1-2026
Abstract
Melanoma, a highly aggressive form of skin cancer, is primarily driven by DNA alterations often linked to environmental factors such as ultraviolet radiation. Addressing the need for improved early detection, this study tackles the key limitations of current methods, which frequently employ convolutional neural networks (CNNs) but struggle with feature selection, class imbalance, hyperparameter tuning, and generalizability. Our strategy leverages dilated convolution (DC) layers trained using reinforcement learning (RL). Unlike other RL-based approaches that handle these challenges in isolation, our method introduces a multi-stage architecture. It integrates RL for feature selection and class balancing. Shapley additive explanations (SHAP) guide feature identification, while augmented rewards for underrepresented classes help mitigate data imbalance. Bayesian optimization hyperband (BOHB) is used for hyperparameter tuning in a unified training process. BOHB combines the predictive strength of Bayesian optimization with the efficiency of hyperband, accelerating model tuning. It also includes an online GAN module for dynamic data augmentation that responds to the evolving output of the RL agent. A novel regularization technique stabilizes GAN training and prevents mode collapse. Importantly, existing RL methods face the challenge of balancing exploration and exploitation. In our RL model, the scope loss function (SLF), integrated with RL, balances exploration and exploitation, thereby ensuring accuracy and generalizability. Collectively, the model jointly tackles four persistent challenges in earlier RL-based approaches: poor exploration–exploitation balance, unstable reward dynamics, static data augmentation, and manual hyperparameter tuning. The model achieved F-measures of 94.3 %, 93.7 %, and 91.5 % on ISIC-2020, HAM10000, and PH2, respectively. This advancement significantly improves early melanoma detection and supports more accurate treatment decisions, contributing valuably to the ongoing effort to combat this lethal cancer.
Keywords
Bayesian optimization, Generative adversarial network, Melanoma, Reinforcement learning, Skin cancer
Publication Title
Biomedical Signal Processing and Control
ISSN
1746-8094
DOI
10.1016/j.bspc.2025.108668
Recommended Citation
Yang, Jing; Wan, Yajie; Diao, Su; Alfarraj, Osama; Alblehai, Fahad; Tolba, Amr; Shaikh, Zaffar Ahmed; Por, Lip Yee; Alizadehsani, Roohallah; and Zhang, Yudong, "Melanoma Detection through Combining Reinforcement Learning, Generative Adversarial Network, and Bayesian Optimization" (2026). Research Publications (2026 to 2030). 1400.
https://knova.um.edu.my/research_publications_2026_2030/1400
Funders
Universiti Malaya, King Saud University (AIM-C09-2025), National Science and Technology Council (NSTC-114-2224-E-027-001, RMF1506-2021)
Volume
112
Publisher
Elsevier Ltd