scMustree: a multi-scale functional hierarchy for single-cell transcriptomic analysis.

Abstract

Resolving cellular heterogeneity requires methods that capture both discrete cell types and their functional relationships. Existing single-cell clustering approaches often produce flat partitions, limiting their ability to reveal rare cell states and continuous biological transitions. Here, we introduce scMustree, a computational framework that constructs a multi-scale hierarchy of single-cell transcriptomes, enabling integrated analysis of cellular organization and functional relationships. scMustree employs a top-down iterative decomposition to isolate transcriptionally homogeneous populations, followed by a bottom-up merging strategy guided by cluster-specific functional rankings-derived from differential expression and an isolation-forest-based scoring mechanism that quantifies functional distinctness. This unified approach captures lineage structures, functional similarities, and transitional states. Across 11 benchmark datasets, scMustree achieves competitive clustering accuracy while offering substantially enhanced biological interpretability. In diverse biological systems, it uncovers biologically consistent hierarchies and identifies previously uncharacterized cell states, including fibroblast and T-cell subsets in cancer and lipid-associated microglial populations in Alzheimer's disease. By integrating structural and functional information, scMustree provides a scalable and biologically grounded framework for multi-resolution exploration of single-cell ecosystems, enabling discovery of rare and disease-relevant cell states across diverse biological contexts. Freely available at Github(https://github.com/xuyp-csu/scMustree) and Zenodo(https://zenodo.org/records/17480562). Supplementary data are available at Bioinformatics online.