Why Empirical Risk Minimization Performs Well for Open Set Domain Adaptation: A Theoretical Analysis From Causal View.

Abstract

Open set domain adaptation (OSDA) faces two critical challenges: the emergence of unknown classes in the target domain and changes in observed distributions across domains. Although numerous studies have proposed advanced algorithms, recent experimental results demonstrate that the classical empirical risk minimization (ERM) approach still delivers state-of-the-art performance. However, few theories can effectively explain this disputed phenomenon. To address the theoretical gap, we focus on constructing a causal theoretical framework for OSDA. We formulate the novel concepts of the fully informative causal invariance model (FICIM) and the partially informative causal invariance model (PICIM). Subsequently, we derive an OSDA theoretical bound to prove that the ERM performs well when the source domain follows FICIM, while it performs poorly when the source domain follows PICIM. The different results may be attributed to the varying amounts of available information when bounding the target domain's stable expected risk. Finally, across different datasets, we conduct extensive experiments on the FICIM and PICIM source domains to validate the effectiveness of our theoretical results. Moreover, our findings can also support the training and fine-tuning of large language models (LLMs).