CocoAdapter: Efficient end-to-end temporal action detection via self-constrained multi-cognitive adapters.

Abstract

End-to-end training in temporal action detection (TAD) has shown great potential for performance improvement by jointly optimizing the video encoder and action classification head. However, memory bottlenecks have limited the performance of end-to-end TAD. To alleviate the memory overhead during training, this paper explores the application of adapters in TAD and proposes a specialized TAD-oriented self-constraint multi-cognitive adapter (CocoAdapter). Based on CocoAdapter, we construct a novel baseline, CocoTad. Our proposed CocoAdapter utilizes self-constraint projection layers to adjust multiple cognitive convolutional groups based on network depth, enabling a fine-tuning process tailored to the TAD task. As a result, the network only needs to update the parameters in CocoAdapter to achieve end-to-end training, significantly reducing memory consumption during training. We evaluate our model on four representative datasets. Experimental results demonstrate that our proposed CocoTad surpasses previous state-of-the-art methods in terms of mAP.