Microsoft Research Reveals Developers Secretly Manage AI Agents Like Managers
A qualitative study of 17 developers reveals that overseeing AI coding agents is a four-part, full-lifecycle job — and that most of it happens before the agent writes a single line of code.
- New qualitative study: Researchers Dhanorkar, Passi, and Vorvoreanu interviewed 17 experienced developers to map how they actually oversee AI coding agents in practice.
- Four forms of oversight: A priori control, co-planning, real-time monitoring, and post hoc review — the first two are proactive, contradicting prior research that focused only on reactive oversight.
- Developers rarely monitor in real time: Most participants never proactively watched agent reasoning traces; they relied on quick eyeballing or skipped monitoring entirely.
- Four coping heuristics found: Developers use the agent's plan as a proxy for its work, treat passing tests as correctness guarantees, skim outputs rather than read them, and trust agents blindly in unfamiliar domains.
- Tooling gap identified: Agent builders lack configuration guides, prompting aids, augmented diff tools, and execution-context visualizations that would make oversight feasible at scale.
- Role shift documented: The study argues the developer role is shifting from "craftsman" to "developer-manager," with oversight becoming the primary competency.
Ask most people what it means to "oversee" an AI agent and they will describe something reactive: watch it run, catch its mistakes, fix the output. A new academic paper by Shipi Dhanorkar, Samir Passi, and Mihaela Vorvoreanu , all affiliated with Microsoft Research , challenges that picture entirely. Their study, accepted to FAccT '26, is one of the first to examine how developers actually oversee software agents in practice, and the reality is messier, more proactive, and far more labor-intensive than existing frameworks suggest.
The gap between theory and the terminal
Existing research on agent oversight is largely conceptual , normative frameworks exist, but how users actually oversee agents is less known. Governance documents like the EU AI Act mandate meaningful human control over high-risk AI, but they assume oversight is feasible without specifying what it looks like on the ground. The researchers set out to fill that gap with real interview data.
Unlike conventional AI, agents can autonomously plan, use external tools, and take actions in real-world environments. The impact of agents is perhaps most evident in software engineering, which has seen a proliferation of tools such as Cursor, Claude Code, Cline, ChatGPT Codex, and GitHub Copilot. But with that power comes a new category of failure: agents that misinterpret requests, lose context mid-task, silently propagate bugs into production, or , in high-profile incidents , erase entire databases.
Who they talked to, and how
Drawing on interviews with 17 experienced developers, the researchers conducted an exploratory inquiry examining what forms of emergent oversight work developers perform, when, and how. Participants were deliberately chosen as power users: 13 had seven or more years of programming experience, and 16 of 17 used agents several times a week for professional work , 12 of them daily. Tools used spanned Cursor, Claude Code, GitHub Copilot, Cline, and Amazon CodeWhisperer, across domains including financial technology, storage systems, ad experimentation, and ML research.
The methodology was qualitative and inductive. Interviews were semi-structured and one hour long, with the first author conducting open coding, then axial coding across 13 themes, and finally selective coding to arrive at the two core findings: the forms of oversight work, and the heuristics developers use to cope. Crucially, the researchers used temporal anchoring , explicitly asking users what they do before, during, and after agent execution , to ground participant responses in end-to-end agent workflows.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
