FLIPS Catches Hidden AI Jailbreaks With 96% Accuracy at ICML
FLIPS fingerprints how an LLM is configured, not just which model it is, enabling regulators to audit deployed AI behavior for the first time
- New paradigm: FLIPS introduces instance-level LLM fingerprinting, identifying how a model is configured, not just which model it is.
- Strong results: 96% closed-set and 90% open-set accuracy across 237 model instances, vs. 35% for the LLMmap baseline.
- Core mechanism: LLMs produce biased (non-random) binary sequences; FLIPS uses NIST statistical tests to turn those biases into a configuration fingerprint.
- Regulatory focus: Designed for compliance auditors who need to verify deployed behavior, not model provenance or IP ownership.
- Covers key config axes: Distinguishes instances varying by system prompt, sampling temperature, and quantization level.
- Open source: Code available at github.com/GurvanR/FLIPS-LLM-Instance-Fingerprinting; accepted at ICML 2026.
Every LLM fingerprinting paper published in the last two years has been solving the wrong problem. They ask: which model is this? But for AI regulators, the real question is: how is this model configured? The same Llama 3 weights, running under a jailbreak system prompt with high temperature sampling, behaves like a completely different system than the safety-aligned default. A new paper accepted at ICML 2026 introduces FLIPS, the first fingerprinting method designed to tell those two apart.
The Gap Between IP Protection and Regulation
The LLM fingerprinting field was born out of intellectual property concerns. The exorbitant cost of training LLMs from scratch makes it essential to fingerprint models to protect intellectual property via ownership authentication and to ensure downstream users comply with license terms. Tools like LLMmap became the state of the art for this: LLMmap sends carefully crafted queries to the target application and analyzes the responses to identify the specific LLM version in use, with query selection informed by domain expertise on how LLMs generate uniquely identifiable responses. With as few as 8 interactions, LLMmap achieves over 95% accuracy in identifying 42 different LLM versions.
But robustness to configuration changes is a feature of IP-protection fingerprinting, not a bug. The whole point is that the fingerprint survives fine-tuning, quantization, and system prompt changes. For a regulator, that is exactly the wrong property. LLMmap is designed to be robust across different application layers, allowing it to identify LLMs operating under various system prompts, stochastic sampling hyperparameters, and even complex generation frameworks such as RAG or Chain-of-Thought. In other words, it deliberately ignores the configuration signal that regulators need most.
Why Configuration Is the Whole Problem
The core insight of FLIPS is that an LLM's behavior is not just a function of its weights. The same base model can be deployed in radically different ways:
- System prompt: a jailbreak prompt can strip safety guardrails entirely
- Sampling temperature
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
