Idiap's VLEED Scrubs Hidden Demographic Data From Face Recognition Vectors
A new post-hoc method strips gender and ethnicity signals from face embeddings using entropy maximization, without retraining the recognition backbone.
ProTakeaways
- VLEED is a post-hoc VAE that removes categorical attributes like gender or ethnicity from pretrained face embeddings.
- It splits embeddings into a 480-dim identity residual and a 32-dim class latent with a class-conditional Gaussian prior.
- Disentanglement works by maximizing the entropy of an auxiliary classifier, equivalent to minimizing mutual information with the attribute.
- A single knob
λ_dissweeps a continuous privacy-utility tradeoff, evaluated on IJB-C, RFW, and VGGFace2. - Linear leakage falls to majority-class baselines quickly; deep MLP leakage requires stronger settings that also hurt verification.
- Code and demographic labels will be released; backbone-agnostic multi-attribute and continuous-variable extensions remain open.
Face recognition models are trained to identify people, but the embeddings they produce carry more than just identity. Anyone with access to those vectors can train a shallow classifier and read off gender, ethnicity, age, and other soft-biometric attributes the subject never agreed to share. A team at the Idiap Research Institute has proposed VLEED (Variational Latent Entropy Estimation Disentanglement), a post-hoc transformation that scrubs a chosen categorical attribute from pretrained embeddings while keeping verification accuracy usable.
Leakage hiding in a single vector
Deep face recognition models learn embeddings that are highly discriminative for identity, but analyses have shown that state-of-the-art models also capture soft-biometric attributes like gender, age, ethnicity, and even transient features such as hairstyle and eyewear, despite never being explicitly trained to predict them. That creates two headaches. First, third parties can infer sensitive attributes the subject never intended to disclose. Second, downstream systems inadvertently rely on demographic signals and produce disparate treatment across protected groups.
Prior fixes have limitations. Linear methods like INLP and IVE project embeddings orthogonally to attribute-predictive directions but operate on point estimates and cannot capture the full distributional structure. Nonlinear autoencoder approaches such as PFRNet and ASPECD try to match statistical moments across demographic groups, yet matching low-order statistics does not guarantee independence, and adversarial methods usually need end-to-end control of the recognition pipeline.