A hierarchical prompt and prototype learning framework for brain disorder classification.

Abstract

Accurate diagnosis of brain disorders (BDs) is challenging in clinical practice. Most existing deep learning-based methods perform diagnosis only in a one-step manner, ignoring the step-wise, multi-level diagnosis processes as performed by radiologists. This oversight often leads to a high risk of misdiagnosis, especially for long-tail or challenging BDs. In this work, we introduce a Hierarchical Prompt and Prototype Learning (HP²L) framework for BD diagnosis, which emulates multi-level diagnostic procedures. HP²L explicitly captures hierarchical relationships among 23 BDs and groups them into three diagnostic levels: coarse classes (e.g., vascular lesions), intermediate classes (e.g., hemorrhage), and fine-grained classes (e.g., chronic hemorrhage). HP²L integrates three key innovations: (1) Hierarchical Prompting Vision Transformer (ViT) backbone, which performs coarse-to-fine feature extraction for step-wise BD classification; (2) Prompt Learning, which employs optimizable prompt tokens that encode diagnostic knowledge, guiding the classification at each level of the hierarchy; (3) Prototype Learning, which enriches the prompt token with BD-specific prototypes by injecting diagnostic information to enhance diagnosis performance. Extensive evaluations on 54,360 subjects across six multi-center datasets show that HP²L consistently outperforms state-of-the-art methods, achieving a balanced accuracy of 88.43% for both common and long-tail BDs, 8.42 percentage points higher than the best-performing benchmark. Furthermore, HP²L improves interpretability by aligning its predictions and attention visualizations with the clinical hierarchical reasoning process. The code and a portion of data (more data will be released after the decision of the paper) are available under: code, data.