Stanford's OctoTools Beats LangChain and AutoGen by 10.6% Without Fine-Tuning
Stanford's training-free agent framework hits 58.5% average across 16 reasoning benchmarks, beating GPT-4o by 9.3% and topping LangChain and AutoGen by up to 10.6%.
ProTakeaways
- Stanford releases OctoTools, a training-free agent framework beating GPT-4o by 9.3% across 16 benchmarks.
- Uses standardized tool cards, a planner agent, and a separate executor agent for reliable tool orchestration.
- Outperforms AutoGen by 10.6%, GPT-Functions by 7.5%, and LangChain by 7.3% with identical tools and budgets.
- Task-specific toolset optimization gives 58.9% average accuracy versus 57.4% when enabling every tool.
- Robust across weaker backbones like GPT-4o-mini and Qwen2.5, degrades gracefully under injected tool failures.
- Open source on GitHub, average query costs $0.05 on GPT-4o.
A team from Stanford has released OctoTools, an open-source agentic framework that wraps large language models with a planner, an executor, and a library of pluggable tool cards. The pitch is simple: no fine-tuning, no bespoke prompt engineering per domain, and better reasoning accuracy than the popular agent stacks. Across 16 tasks including MathVista, MMLU-Pro, MedQA, and GAIA-Text, it delivers an average accuracy gain of 9.3% over GPT-4o, and beats AutoGen, GPT-Functions, and LangChain by up to 10.6% when given the same set of tools.
Why bolt-on tool use keeps breaking
Getting an LLM to reliably call external tools sounds easy until you try to do it across domains. Most existing systems fall into one of two buckets. Some rely on heavy fine-tuning with curated tool-use data, which locks them into a narrow set of skills. Others target a single domain like chemistry or web browsing, so switching tasks means switching frameworks. General agent libraries like LangChain and AutoGen are more flexible, but the authors argue they emphasize high-level abstractions or multi-agent collaboration rather than complex reasoning, and existing methods remain restricted to specialized domains, limited tool types, or extra training data.
OctoTools targets the middle ground: a training-free system where adding a new tool is as light as writing a short metadata file, and where planning is decoupled from code execution so errors in one stage do not poison the other.
Tool cards, a planner, and an executor walk into a loop
The framework has three moving parts working over a shared context that grows step by step.

- Tool cards are standardized wrappers around any tool: a Python calculator, a Google Search API, an image captioner, a pathology classifier. Each card carries the tool's input and output schema, example commands, and even soft guidance like limitations and best practices that the planner can read.