Collaborative Hyperparameter Recommendation by Coupled Matrix Factorization With Kernel.

Abstract

Hyperparameters significantly influence the learning process and performance of machine learning algorithms, rendering their effective selection a critical challenge. Meta-learning-based hyperparameter recommendation has shown promise, yet existing methods typically rely solely on either meta-features or partial hyperparameter evaluations, limiting their flexibility and real-world applicability. To overcome these constraints, we propose a novel hyperparameter recommendation framework based on coupled matrix factorization (CMF). Our approach flexibly integrates meta-features and partial hyperparameter evaluations, either individually or concurrently, through a dual-component model: a matrix completion module leveraging partial hyperparameter evaluations and an auxiliary module utilizing meta-features. By sharing a common factor matrix, this design captures nuanced patterns in metadata that single-method approaches often miss. Additionally, we incorporate kernel-based techniques to address nonlinearity in the metadata, further enhancing recommendation accuracy. We evaluate our method using well-established classifiers, including support vector machines (SVMs) and Vision Transformers (ViTs), as backbone algorithms. Extensive experiments on hundreds of real-world datasets, assessed across multiple metrics, demonstrate that our approach outperforms existing meta-learning-based recommendation models and search algorithms.