Skip to main content
Back

Karpathy's Overnight AI Research Loop Gets an Open-Source Home in pi-autoresearch

A new pi extension generalizes Karpathy's autoresearch loop to any optimization target you can score, from bundle size to LLM training.

Repo, Full Access
  • pi-autoresearch is a new extension for the pi terminal coding agent that runs autonomous optimization loops.
  • Inspired by karpathy/autoresearch, generalized to any metric you can score from a command.
  • Ships tools, a /autoresearch slash command, a live dashboard, and skills to set up and finalize sessions.
  • Uses Median Absolute Deviation to compute a confidence score that flags gains inside benchmark noise.
  • Session state lives in a .auto/ folder so agents can resume across restarts and context compaction.
  • MIT licensed, installed via pi install npm:pi-autoresearch, with maxIterations to cap token spend.

The autonomous experiment loop that Karpathy popularized for overnight nanochat tuning has a general-purpose home. pi-autoresearch is a new extension for pi, an open-source terminal coding agent, that hands the model a full try-measure-keep-or-revert workflow for any command you can benchmark. Try an idea, measure it, keep what works, discard what doesn't, repeat forever.

The extension is inspired by karpathy/autoresearch, the repo where an AI agent is handed a small but real LLM training setup and left to experiment overnight: modify code, train for five minutes, check if the result improved, keep or discard, repeat. pi-autoresearch takes that pattern and makes the target arbitrary. Runtime, bundle size, build times, Lighthouse scores, validation loss on a training run, anything you can reduce to a number.

Wiring the loop

The design separates infrastructure from domain knowledge. The extension exposes three tools the agent calls directly:

  • init_experiment: one-time session config for name, metric, unit, and direction (higher or lower is better)
  • run_experiment: runs any command, times wall-clock duration, and captures output
  • log_experiment: records the result, auto-commits, and updates the widget and dashboard