From heatmaps to language: Explainable skin cancer diagnosis using CNN-VLM integration with an expert-annotated dataset.

Abstract

Skin cancer diagnosis relies on visual interpretation of lesion morphology, yet most deep learning models function as black boxes with limited clinical interpretability. This study presents an explainable diagnostic framework that integrates a fine-tuned Xception network with a Convolutional Block Attention Module (CBAM) for skin lesion classification and a domain-adapted vision-language model (VLM) for generating clinically meaningful explanations. The classifier was trained on a preprocessed and augmented HAM10000 dataset, while Grad-CAM++ was employed to highlight discriminative lesion regions. These heatmaps were manually annotated with expert morphological descriptions to construct a novel image-text corpus comprising 347 samples, which was subsequently used to parameter-efficiently fine-tune MedGemma-4B using Low-Rank Adaptation (LoRA). The proposed classifier achieved a test accuracy of 85.66% with an AUC of 0.9392 across seven lesion classes. Furthermore, the fine-tuned VLM improved the average explanation quality score from 62.14 to 87.86, representing a 25.72-point gain over the baseline according to automated LLM-based evaluation. The resulting framework provides clinicians with classification predictions, visual attention maps, and expert-aligned textual rationales, offering a practical step toward trustworthy and explainable AI for dermatological diagnosis.