HKUST's ContextLens Makes GPT-4o-mini Beat Fine-Tuned Models on GDPR Compliance
A new semi-rule-based framework tackles the hidden weakness in LLM compliance judges: real-world context is almost always incomplete or ambiguous.
Takeaways
- ContextLens is a semi-rule-based framework that decomposes GDPR and EU AI Act into a four-level priority hierarchy for LLM compliance judgment. Paper
- Instead of asking for a verdict, it prompts models with narrow sub-rule questions that allow "not sure" and "none of the above" answers.
- GPT-4o-mini with ContextLens beats fine-tuned ContextReasoner-7B-PPO, with average accuracy of 89.99% versus 88.25%.
- Llama-3.1-8B-Instruct gains almost 15 accuracy points over direct prompting without any additional training.
- Analysis reveals nearly 100% of privacy compliance benchmark cases contain more than five unknown contextual factors.
- Code and data are available on GitHub; large reasoning models like DeepSeek-R1 do not benefit due to overthinking.
When researchers benchmark large language models as compliance judges for regulations like GDPR or the EU AI Act, they usually hand the model a tidy paragraph of context and ask whether it violates the law. A new paper called ContextLens argues that this setup is quietly broken. Real privacy and safety scenarios are messy, missing key facts, and full of ambiguity, and treating them as complete is exactly why LLM judges get things wrong.
The team from HKUST, Beihang, NUS, and the University of Macau built a semi-rule-based framework that forces LLMs to reason through legal statutes the way lawyers do, one hierarchical chunk at a time, while explicitly flagging what the context does not tell them. Code and data are public, and the results are strong enough that a well-prompted GPT-4o-mini beats fine-tuned reasoning models on the same task.
Why perfect-context assumptions break in the wild
The motivating example is sharp. If a user consents to ChatGPT's terms and conditions, most LLM judges will call this compliant. But if that user is a child under 13 in the EU, GDPR requires a legal guardian's authorized consent, and the same scenario flips from lawful to unlawful. The context is not wrong, it is incomplete, and the model has no mechanism to notice.
Existing safety practices lean on regular-expression filters and safety-alignment training, both of which miss the full range of contextualized concerns and generalize poorly to evolving attacks. More recent work uses LLM-as-a-judge with contextual integrity theory, but almost every one of those methods assumes the input already contains everything needed to decide.