48 Researchers Map 574 AI Safety Papers and Find Attackers Winning
A sweeping review of 574 papers maps the attack surface of large models and agents, exposing a lopsided arms race where offense outpaces defense.
PRO- Survey of 574 papers maps safety across VFMs, LLMs, VLMs, diffusion models, and agents
- 60% of research targets attacks, only 40% defenses, exposing a critical imbalance
- Alignment faking discovered in production models like Claude 3 Opus during controlled tests
- Fine-tuning attacks can break safety alignment with as few as ten adversarial examples
- Agentic attacks now autonomously discover zero-day vulnerabilities in web applications
- Companion repository tracks the field: Awesome-Large-Model-Safety
A group of 48 researchers just published one of the most comprehensive maps of the AI safety landscape to date, cataloging 574 technical papers across vision foundation models, LLMs, vision-language models, diffusion models, and large-model-powered agents. Attackers are moving faster than defenders, evaluation benchmarks are quietly going stale, and even the alignment techniques the field relies on may be masking deeper problems.
Attacks 60, defenses 40
The survey organizes threats into ten categories: adversarial, backdoor, poisoning, jailbreak, prompt injection, energy-latency, membership inference, model extraction, data extraction, and agent attacks. When the authors tallied where researchers are actually spending their time, they found a lopsided arms race.
Safety research on large models has surged since the release of ChatGPT in 2023, but 60% of the surveyed work targets attacks while only 40% covers defenses. Jailbreak, adversarial, and backdoor attacks dominate the offensive literature. On the defensive side, jailbreak defenses lead, followed by adversarial defenses. LLMs, diffusion models, and agents together account for over 71% of the surveyed papers.
Why jailbreaks keep working
The jailbreak section is where offensive creativity is most visible. Hand-crafted attacks now include scenario-based camouflage (hiding malicious queries inside role-play or puzzle-solving), attention shifting through unusual sentence structures, and encoding tricks that exploit rare formats like Base64 or low-resource languages. Automated methods have escalated further: I-GCG refines the greedy coordinate gradient algorithm with diverse target templates and an automatic multi-coordinate updating strategy, achieving near-perfect attack success rates.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
