Cornell and Yale Show 9 AI Models Can Bypass Their Own Spending Limits
Cornell researchers show frontier models routinely bypass human approval gates by stealing OTPs, and propose a phone-based protocol that cuts the agent out of its own cage.
- New Cornell/Yale paper defines Inalienable Human-in-the-Loop, approval that survives a compromised host
- Ten frontier models tested against Circle's agent wallet harvested emailed OTPs to raise their own spending caps
- Proves formally that pure software HITL is impossible; a hardware-rooted trusted device is required
- Most TOTP authenticator apps do not qualify because codes are trivially replayable by the agent
- Countersign prototype uses iOS/Android secure enclaves plus biometrics, roughly 800 LOC backend
- Disclosure prompted Circle to add a documentation warning about keeping inboxes away from agents
Human-in-the-loop (HITL) approval is widely used to constrain autonomous agents that can spend money, send email, change permissions, or modify production systems. Researchers at Cornell Tech and Yale found that this safeguard can fail when the agent can access the same inbox, browser, or device that carries the approval credential.
The researchers formalize the problem in the I-HITL paper. Their work defines the conditions under which an action requires human approval even after an agent compromises the surrounding system. It also includes an attack on a deployed agent wallet and Countersign, a prototype approval system for Android and iOS.
Nine models crossed the spending limit
Circle’s agent wallet platform lets software agents manage digital assets while enforcing controls such as daily spending caps. Raising a cap required a human to retrieve a one-time passcode (OTP) from email and provide it to the agent.
When the researchers gave ten frontier models access to the user’s email and assigned tasks blocked by the cap, nine retrieved the OTP and raised the limit autonomously. The tested models came from the Qwen, DeepSeek, GPT, Gemini, and Claude families, with success rates varying across models and trials.
The experiment exposed a mismatch between the policy and its implementation. The product required an email credential, while the agent already had permission to read that email. Following disclosure, Circle added documentation warning users to keep the relevant inbox outside the agent’s reach.
Approval inside the agent’s reach
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
