CASIA's Direct Discrepancy Replay Beats Deepfake Detectors While Ditching Stored Faces

Deepfake generators keep multiplying, and every new family of forgeries poses the same headache for detectors: how do you teach a model the new manipulation without erasing what it learned about the old ones? A paper from researchers at CASIA and UCAS proposes a cleaner answer than the usual approaches, and it comes with a useful side effect for privacy.

The work introduces Direct Discrepancy Replay, a continual learning framework for face forgery detection that stops storing past face images entirely. It condenses the statistical gap between real and fake faces into a tiny bank of learnable maps, then reconstructs training examples on the fly by combining those maps with real faces from the current task.

Why detectors forget

Continual Face Forgery Detection (CFFD) is the task of updating a detector as new manipulation techniques appear. Continual learning has become the go-to paradigm here because it lets models acquire new knowledge while retaining previously learned capabilities, typically through sample replay, parameter regularization, or gradient projection. Rehearsing a few examples from earlier tasks during new training, known as replay, is the dominant strategy.

Both flavors of replay have real drawbacks:

• Selection-based replay keeps a small subset of historical images. Under tight memory budgets it misses the diversity of past forgery cues, and it retains actual identifiable faces in memory.
• Perturbation-based replay, like the Historical Distribution Preserving (HDP) approach that uses universal adversarial perturbations to simulate past forgeries, avoids storing raw faces but stays tied to the previous decision boundary and drifts as the backbone updates.