Anthropic Lets Outside Researchers Study 250,000 Real Claude Conversations

For the first time, an AI lab has let outsiders run their own studies on its production traffic. Anthropic just published the results of a pilot in which three external research groups designed independent studies against roughly 250,000 real Claude.ai and Claude Code conversations, using the same privacy-preserving analysis tool Anthropic's own teams use internally.

The partners were the Social and Language Technologies (SALT) Lab at Stanford, the Human Information Processing Lab at the University of Oxford, and METR, a non-profit that evaluates frontier AI models. Each group wrote its own research questions and analyzed conversations sampled from April to May 2026. Anthropic ran the queries on their behalf but, per the contract, had review rights limited to user privacy, potential policy violations, its own confidential information, and research accuracy, with no say over the findings themselves.

Why an outside view matters

Until now, anyone curious about how people actually use frontier AI had two bad options: rely on analyses labs publish, which reflect real usage but answer the lab's questions, or use public datasets, which you can study freely but skew toward casual use. Independent groups like MIT's AI Observatory have tried to fill that gap by stitching together open datasets, but the scale is orders of magnitude smaller than what labs see internally.

The mechanism here is Anthropic Insights, formerly known as Clio. A researcher writes a question like "What type of guidance is this person asking for?" and Claude answers it for every conversation in the study, with answers aggregated into categories. Researchers only see the final categories and the percentage of conversations that fall under each, never the underlying chats.

What the Stanford team found

SALT's writeup is the first one public, and it upends a common assumption about how people delegate to AI. Highlights:

• Over half of Claude conversations involved people delegating consequential tasks, meaning work that affects others or is hard to undo, contradicting prior research suggesting people mostly keep such work for themselves.
• People were most likely to bring high-stakes work to Claude when seeking professional guidance, particularly on legal or financial questions.
• In nearly three-quarters of conversations, people set the direction while Claude assisted, and they usually adapted its output rather than using it verbatim.
• Friction in the collaboration was often productive: iterating on unclear requests pushed people to clarify their intent and refine outputs.

Early hints from Oxford and METR

The other two studies are still in progress, though preliminary findings are worth flagging. Oxford's HIP Lab looked at the emotional side of AI use and found that a warm Claude tracked with more positive users, refusals and disagreement drew pushback, and eccentric responses correlated with users getting more intellectually engaged. The patterns of absorption, frustration, and enjoyment in Claude conversations also closely resemble those documented in a separate study on everyday internet browsing.

METR is running the study developers will care about most: quantifying real productivity gains from coding agents. Preliminary findings indicate newer models deliver significant speedup over older ones, and Claude's own estimates of how long tasks would take correlated with actual completion times from a prior developer study, which validates using the model as a judge for time-savings analysis. METR next wants to measure how much AI accelerates research itself, a figure that gets more consequential as models take on more of their own development.

The engineering catch

Anthropic is candid that the pilot was harder to run than expected. Because researchers never see raw conversations, the tool is sensitive to question wording, and a poorly phrased query can place chats into misleading buckets that no one can catch by reading the source. Internally, Anthropic irons this out by iterating for weeks; externally, that was infeasible because each dataset requires privacy review. Their workaround was to have partners prototype questions on WildChat, a public human-AI conversation dataset, but WildChat skews casual and creative and didn't always transfer to real Claude traffic.

On privacy, Anthropic ran an additional third-party audit of all data shared with the researchers, conducted by Imperial College London, and is publicly releasing the aggregate datasets from each project.

What now

Anthropic is soliciting proposals from other researchers who want access to Anthropic Insights for work they cannot do today. The bar is that a study needs to be something the aggregate tool can answer without requiring iteration on raw conversations, and the lab is explicit that scaling this program will be slow because of the privacy and review overhead. If it holds up, this is the first credible template for independent oversight of what people actually do with frontier AI, rather than what labs choose to publish about it.