Skip to main content
Back

DP-MacAdam Beats Every Private AI Optimizer Without Manual Tuning

DP-MacAdam unifies adaptive gradient clipping and Adam-style momentum under differential privacy, beating DP-SGD, AdaClip, and DP-Adam without manual threshold tuning

Paper, Full Access
  • New optimizer: DP-MacAdam unifies adaptive gradient clipping (AdaClip) and Adam-style momentum under a single set of gradient statistics.
  • Core fix: Both AdaClip and DP-Adam independently estimate gradient mean and variance -- DP-MacAdam computes them once and reuses them for both tasks.
  • Bias correction: A novel correction factor removes the DP noise inflation bias from the variance EMA, fixing the known DP-Adam degradation problem.
  • Results: Outperforms DP-SGD, AdaClip, and DP-Adam on MNIST and CIFAR-10 across multiple privacy budgets (epsilon values), with no manual clipping threshold tuning.
  • No code release: No public implementation is available yet; practitioners must implement from the paper's algorithm description.
  • Limitation: Experiments limited to MNIST and CIFAR-10; performance on large-scale models (LLMs, ViTs) is untested.

DP-MacAdam is a new differentially private optimizer that solves a frustrating inefficiency sitting at the heart of private machine learning: two separate lines of research had each independently figured out how to estimate gradient statistics under privacy constraints, but neither had thought to share those estimates across both tasks. The result was wasted signal, redundant computation, and suboptimal models. DP-MacAdam fixes that by doing both jobs with one set of numbers.

Why private training is still painful

Differentially private stochastic gradient descent (DP-SGD) is the most widely used method for training machine learning models with provable privacy guarantees. The core idea is simple: clip individual gradients to a maximum Euclidean norm and add random noise to the average of a batch of gradients. Clipping bounds the individual contribution any sample can make to the model parameters, and adding random noise obfuscates the contributions of any individual example.

But that clipping threshold is a landmine. A key challenge in DP-SGD is setting the per-sample gradient clipping threshold, which significantly affects the trade-off between privacy and utility. Set it too low and you throw away useful gradient signal. Set it too high and you have to inject more noise to maintain the same privacy guarantee, which degrades model performance. The optimal threshold can vary over the course of training, across tasks, and between datasets, limiting the effectiveness of a fixed setting.

The community has attacked this from two directions, but until now those directions never merged:

  • Adaptive clipping (AdaClip): AdaClip shifts and scales the gradient prior to clipping and adding noise so that the clipped gradient yields a more informative descent direction. The shift and scaling parameters are selected adaptively based on the empirical mean and variance.
  • Adaptive momentum (DP-Adam): DP-Adam exploits Adam-like momentum updates based on the gradient mean and variance to accelerate training, but does not exploit these estimates for adaptive clipping.

Both algorithms independently compute running estimates of the gradient mean and variance. They just never shared them. That is the gap DP-MacAdam closes.