Adaptive differential privacy mechanism for enhanced deep learning model utility and privacy.

Abstract

The potential leakage of training data privacy in deep learning has been a topic of concern for researchers and the public. To provide formal and rigorous privacy guarantees, many learning systems integrate differential privacy (DP) with SGD optimizer to train models. To achieve private SGD, existing algorithms often limit the sensitivity by clipping the gradient using a constant, which can significantly impact model performance and requires careful tuning. Furthermore, a fixed privacy budget allocation limits the trade-off between model utility and privacy protection. To address this, we propose a novel deep learning strategy with adaptive DP mechanism, which achieves both adaptive sensitivity and adaptive privacy budget allocation. Specifically, during the training process, we cluster the average gradients of batch samples and adaptively allocate privacy budgets based on the gradients of each cluster. Additionally, we couple the gradient clipping parameters with the learning rate to achieve adaptive gradient clipping, avoiding the typical hyperparameter tuning process associated with constant clipping. We provide a rigorous theoretical privacy analysis demonstrating that the proposed method strictly satisfies DP. Through extensive experimental evaluations, we show that this method outperforms state-of-the-art approaches across multiple mainstream visual tasks.