Skip to main content
Back

MIT's DAT Framework Hits FID 3.29 on ImageNet, Matching Top Diffusion Models

Dual Adversarial Training replaces the unstable Langevin sampler behind Joint Energy-Based Models, scaling to ImageNet 256 while unifying robust classification, generation, and counterfactuals.

Pro

  • DAT replaces SGLD sampling in Joint EBMs with PGD-based adversarial training and BCE loss for stable energy learning.
  • First MCMC-style EBM to scale to ImageNet 256x256, hitting FID 3.29 with ConvNeXt-Large.
  • Single model handles robust classification, generation, and counterfactual explanations without separate training pipelines.
  • Two-stage training lets practitioners start from pretrained adversarially robust classifiers across architectures.
  • Code released at github.com/xuwangyin/DAT, accepted to ICLR 2026.
  • Trade-off: model is overconfident on CIFAR-100 and ImageNet, hurting calibration for downstream probability use.

A single neural network that classifies images robustly, generates high-fidelity samples, and produces plausible counterfactual explanations has been a longstanding target for hybrid modeling. A new ICLR 2026 paper from researchers at MIT and an independent collaborator argues that the biggest obstacle was the sampler, not the idea itself. By swapping the notoriously fragile Langevin dynamics used in Joint Energy-Based Models for an adversarial training loop, they get a hybrid model that finally scales past toy resolutions.

The framework is called Dual Adversarial Training (DAT). It replaces SGLD-based EBM learning with adversarial training, resolving the stability issues that have long limited joint energy-based models. DAT integrates adversarial training for both the discriminative and generative components inside a JEM architecture, reaching competitive results across classification, robustness, and generation on CIFAR-10/100 and ImageNet 256, while producing faithful counterfactual explanations.

Why JEM stalled at CIFAR

The starting point for this line of work is the 2020 result that your classifier is secretly an energy-based model and you should treat it like one. Joint Energy-Based Models (JEM) reinterpret the logits of a standard classifier as an energy function, giving you a generative model for free. In principle, one network could handle both discrimination and generation.

In practice, training that energy function requires drawing samples from it, and the standard tool is Stochastic Gradient Langevin Dynamics (SGLD), a noisy gradient-based MCMC sampler. Training tends to diverge, sample quality lags dedicated generative models, and the whole setup has struggled to scale beyond 32x32 images.