MB-GLOM: An attentive GLOM with multi-head projection and bottleneck residual.

Abstract

The GLOM model, through its unique hierarchical representation design, effectively captures the relationships between local features and global structures within images, making it a prominent focus in the study of interpretable neural networks. However, the conventional contrastive projection strategy used in GLOM incurs high computational costs and exhibits low parameter efficiency, limiting its practical applicability in real-world scenarios. To address this issue, we propose MB-GLOM, an optimized model derived from the original GLOM architecture, which aims to reduce computational overhead while maintaining or even enhancing model performance. Common optimization methods fail to meet GLOM's requirements for global-local feature aggregation, thereby undermining its representational learning capability. Therefore, to effectively streamline GLOM, MB-GLOM employs multi-head projection to refine the traditional contrastive learning head, and optimizes information routing via bottleneck residual connections. Additionally, an adaptive hybrid attention mechanism is introduced to enhance feature expressiveness. Experimental results on several benchmark datasets, including SmallNORB, MNIST, FashionMNIST, CIFAR-10, CIFAR-100, and Tiny-Imagenet demonstrate that MB-GLOM achieves comparable or superior performance to GLOM, while significantly reducing parameter count and computational resource usage. Moreover, the visualization experiment results show that while reducing the computational cost, the MB-GLOM model still maintains excellent hierarchical relationship representation capabilities.