Selective inference for <math xmlns="http://www.w3.org/1998/Math/MathML"><mi>k</mi></math>-means clustering.
Where this comes from
- Record sourced from PubMed, PMID 38264325.
- Also identified by PMC identifier 10805457.
- Licence recorded as CC BY.
- The licence permits redistribution, so the abstract is shown in full and the full text is available from the publisher.
Abstract
We consider the problem of testing for a difference in means between clusters of observations identified via <math xmlns="http://www.w3.org/1998/Math/MathML"><mi>k</mi></math>-means clustering. In this setting, classical hypothesis tests lead to an inflated Type I error rate. In recent work, Gao et al. (2022) considered a related problem in the context of hierarchical clustering. Unfortunately, their solution is highly-tailored to the context of hierarchical clustering, and thus cannot be applied in the setting of <math xmlns="http://www.w3.org/1998/Math/MathML"><mi>k</mi></math>-means clustering. In this paper, we propose a p-value that conditions on all of the intermediate clustering assignments in the <math xmlns="http://www.w3.org/1998/Math/MathML"><mi>k</mi></math>-means algorithm. We show that the p-value controls the selective Type I error for a test of the difference in means between a pair of clusters obtained using <math xmlns="http://www.w3.org/1998/Math/MathML"><mi>k</mi></math>-means clustering in finite samples, and can be efficiently computed. We apply our proposal on hand-written digits data and on single-cell RNA-sequencing data.