CoLA Exposes How AI Data Pruning Leaks Private Info With 97% Accuracy
A new attack framework shows that filtering datasets before training exposes both included and excluded samples, expanding the privacy attack surface across the entire ML pipeline.
PRO- CoLA is a new attack framework showing subset training amplifies rather than reduces privacy risks.
- Introduces Selection-participation MIA, exposing whether a sample entered the pipeline even if excluded from training.
- Exploits inclusion stability: samples consistently reselected across overlapping windows reveal their membership status.
- Reaches ~97% AUC on SP-MIA at 0.6 selection ratio versus ~53% for LiRA on vision models.
- Black-box variant on Pythia-160M exceeds 80% AUC without any knowledge of the selection process.
- Tested across nine pruning methods, ResNet/VGG vision models, and Pythia/GPT-Neo language models on MIMIR data.
The standard playbook for training modern models includes an obvious efficiency trick: don't train on all your data. Coreset selection, dataset pruning, deduplication, and quality filtering are now routine steps that trim datasets down to a manageable, high-quality subset before a single gradient step is taken. The comforting side effect, most practitioners assume, is that fewer training samples means less memorization and therefore less privacy leakage.
A new paper titled CoLA: A Choice Leakage Attack Framework argues that this intuition is dangerously wrong. The authors take the first step toward systematically understanding the privacy risks of subset training and demonstrate that the very choices made during selection can become exploitable signals, exposing both included and excluded data to membership inference.
The blind spot in the data pipeline
Membership inference attacks (MIAs) traditionally ask a single question: was this exact sample in the model's training set? The classical attacker exploits overfitting, since trained samples usually get higher confidence than unseen ones. Subset training complicates that clean split by producing three groups instead of two: included data that trained the model, excluded data that was collected and evaluated by the selector but discarded, and outside data that never entered the pipeline at all.
That third bucket, the excluded data, is where things get uncomfortable. The paper defines two distinct privacy surfaces:
- Training-membership MIA (TM-MIA): the classical formulation. A sample is a member only if it was in the training set.
- Selection-participation MIA (SP-MIA): a broader definition where a sample counts as a member if it participated in the selection process at all, even if it was ultimately thrown out.
SP-MIA is the more disturbing threat, enlarging the notion of membership from model training to the entire data-model supply chain. Knowing that someone's record was in a candidate pool for, say, an HIV screening dataset or a credit pre-approval list is itself sensitive information, regardless of whether the final model was trained on it.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
