Model interpretability enhances domain generalization in the case of textual complexity modeling.

Abstract

Balancing prediction accuracy, model interpretability, and domain generalization (also known as [a.k.a.] out-of-distribution testing/evaluation) is a central challenge in machine learning. To assess this challenge, we took 120 interpretable and 166 opaque models from 77,640 tuned configurations, complemented with ChatGPT, 3 probabilistic language models, and Vec2Read. The models first performed text classification to derive principles of textual complexity (task 1) and then generalized these to predict readers' appraisals of processing difficulty (task 2). The results confirmed the known accuracy-interpretability trade-off on task 1. However, task 2's domain generalization showed that interpretable models outperform complex, opaque models. Multiplicative interactions further improved interpretable models' domain generalization incrementally. We advocate for the value of big data for training, complemented by (1) external theories to enhance interpretability and guide machine learning and (2) small, well-crafted out-of-distribution data to validate models-together ensuring domain generalization and robustness against data shifts.