Cognition's Devin Backs Its $10M Guarantee With a Real Engineering-Hours Meter
Cognition will fund Devin usage up to $10M if its agent fails to deliver enough engineering value, backed by a new hours-equivalent estimator.
- Cognition launched an AI Productivity Guarantee: Devin credits up to $10M if value falls short of spend.
- An estimator agent scores each Devin session for usefulness and equivalent human engineering hours.
- System achieves r_log of 0.74 on 233 held-out sessions, validated against 126 enterprise users.
- Lines-of-code baseline scored just 0.27, confirming code volume is a weak proxy for effort.
- Outperforms Anthropic's Jira-based estimator (0.46) using fuller session traces and DeepWiki context.
- Calibration deliberately underestimates output by 1.4x to keep the guarantee conservative for customers.
Cognition is putting money behind a claim almost no AI vendor has been willing to make: that its agent actually pays for itself. The company behind Devin just announced an AI Productivity Guarantee for enterprise customers, promising that if Devin delivers less engineering value than a customer pays for over the course of an annual contract, Cognition will fund usage with credits up to $10 million until the numbers balance out.
The pitch, written by CEO Scott Wu, is aimed squarely at the dashboards CTOs have been staring at all year. Tokens consumed, lines of code, PRs opened, none of these answer whether the spend produced anything useful. Cognition is trying to replace that activity-metric culture with a single dollar-denominated number: how many engineer-hours of real work did the agent deliver?
Turning agent sessions into hours
The guarantee is backed by a technical system Cognition built to estimate the productive engineering hours behind every completed Devin session. An estimator agent runs over each session, first classifying whether it produced useful output, then estimating how long a human engineer would have taken to produce the same work. The agent gets the user prompt, the resulting PR (if any), the full execution trace of every action Devin took, and codebase context pulled from DeepWiki.
Filtering for useful work matters a lot. If a session created PRs and none were merged, the hours are thrown out. For sessions without a PR, a classifier removes 1 to 20 percent of runs depending on the customer, while keeping legitimately productive non-code work like security scans, bug triage, dependency cleanup, and analytics queries.
The prompt engineering behind the estimator is interesting on its own. It is told to reason about the path a human would have taken rather than the agent's actual trajectory, since agents take detours, recover from environment failures, and produce summary reports a solo engineer would skip. It only credits work the user did not already specify, so a fully written implementation plan counts for less than a bare bug report. And it conservatively assumes the reference engineer already has the cross-disciplinary expertise the task demands, even though Devin's biggest pitch is letting a backend engineer ship frontend or data-science work they could not have done alone.
How accurate is it
Cognition validated the estimator against ground truth from 126 users across enterprise customers, who reviewed 258 of their own sessions and gave time estimates. On a held-out set of 233 sessions, the model hits an r_log of 0.74, with about half of all sessions landing within a factor of two of the human estimate. Individual predictions are noisy, with 2x to 3x errors common in either direction, but Cognition argues the errors are roughly unbiased and cancel out at deployment scale.
To stay on the safe side, they deliberately leave a known bias in place. After log-space calibration, the sum of human estimates is still 1.4x the sum of model estimates, a quirk of summing log-unbiased predictions in linear space. Rather than correct it, Cognition reports the lower number, meaning the guarantee is built on top of a meter that systematically understates Devin's output.
They also compared the full estimator against simpler alternatives, and this is where the methodology earns its keep.

Lines changed alone scores an R²_log of just 0.27, confirming that code volume is a terrible proxy for engineering effort. An estimator that sees only the agent's edit tool calls climbs to 0.45. The full estimator with user messages, trace, and codebase context reaches 0.56. The gap is the investigation, diagnosis, environment setup, and tradeoff reasoning that never shows up in a diff.
Where this sits next to other attempts
Two recent studies tried similar things. METR used GPT-4o and GPT-5 on compressed Claude Code transcripts from seven of its own technical staff and reached r_log of 0.83 on 34 sessions, a higher number Cognition attributes to a more homogeneous user pool. Anthropic ran Claude over 1,000 open-source Jira tickets using only title and description and got r_log of 0.46, below the 0.67 that human developers managed on the same tickets. Cognition's 0.74, on a more diverse enterprise dataset with full session traces, sits in between and is the first such system the company says is running in production.
The catch worth reading
The guarantee is real money, but it has guardrails. Hours are converted to dollar value using a standard global engineering rate and compared against actual consumption near the end of a customer's annual contract. Shortfalls are paid out as credits, not cash refunds, capped at $10M. Eligibility is gated by Cognition's enterprise engagement model, which includes embedded customer teams, fine-grained spend controls, and workshops on prompting fleets of agents productively, all of which conveniently raise the chance the meter clears the bar.
The limitations section of the writeup is candid about what the estimator does not measure. Hours are engineering capacity, not business value, so an hour spent on a canceled project counts the same as an hour fixing a production outage. Quality is not modeled either: if Devin merges a subtle bug that costs a week to debug, the meter still credits the original session. And the ground truth is self-reported by users who knew they were talking to Cognition.
Still, this is a meaningful move in a market where the dominant business model is per-token billing with no accountability for output. If the rest of the industry follows, the negotiation between AI vendors and enterprise buyers stops being about consumption ceilings and starts being about delivered work, which is a much harder thing to fake.