Karpathy's Overnight AI Research Loop Gets an Open-Source Home in pi-autoresearch
A new pi extension generalizes Karpathy's autoresearch loop to any optimization target you can score, from bundle size to LLM training.
Takeaways
- pi-autoresearch is a new extension for the pi terminal coding agent that runs autonomous optimization loops.
- Inspired by karpathy/autoresearch, generalized to any metric you can score from a command.
- Ships tools, a /autoresearch slash command, a live dashboard, and skills to set up and finalize sessions.
- Uses Median Absolute Deviation to compute a confidence score that flags gains inside benchmark noise.
- Session state lives in a .auto/ folder so agents can resume across restarts and context compaction.
- MIT licensed, installed via
pi install npm:pi-autoresearch, with maxIterations to cap token spend.
The autonomous experiment loop that Karpathy popularized for overnight nanochat tuning has a general-purpose home. pi-autoresearch is a new extension for pi, an open-source terminal coding agent, that hands the model a full try-measure-keep-or-revert workflow for any command you can benchmark. Try an idea, measure it, keep what works, discard what doesn't, repeat forever.
The extension is inspired by karpathy/autoresearch, the repo where an AI agent is handed a small but real LLM training setup and left to experiment overnight: modify code, train for five minutes, check if the result improved, keep or discard, repeat. pi-autoresearch takes that pattern and makes the target arbitrary. Runtime, bundle size, build times, Lighthouse scores, validation loss on a training run, anything you can reduce to a number.
Wiring the loop
The design separates infrastructure from domain knowledge. The extension exposes three tools the agent calls directly:
init_experiment: one-time session config for name, metric, unit, and direction (higher or lower is better)run_experiment: runs any command, times wall-clock duration, and captures outputlog_experiment: records the result, auto-commits, and updates the widget and dashboard