Self-Improving AI Skills Learn in the Notebook, Not the File

Hand-authoring agent instruction files is a losing game. Your files rot the moment your underlying tasks shift.

The standard industry fix is skill evolution: rewriting an agent's instruction files from rollout evidence instead of manually updating them.

But when teams build these evolution loops, they almost always forget a critical component: memory of what didn't work.

If your loop has nowhere to store a rejected idea, you are paying full computational price to rediscover every dead end.

WikiSkill, a new framework by Google Research and Virginia Tech, proves that your skill file is actually the least interesting thing your training loop produces.

The paper is "WikiSkill: Compiling Agent Experience into Persistent Knowledge for Skill Evolution", August 2026.

WikiSkill builds a loop where a coach program maintains a private, persistent notebook of every past failure and workaround.

By notebook I mean the loop's private memory which is everything the coach has learned, stored where the working agent cannot see it.

It has two halves. One collects patterns: every failure mode and workaround the coach notices, written down once and reused. The other is a log of every skill change ever tried, with its test score and whether it survived.

When the working agent was strictly forbidden from reading this notebook, accuracy climbed 15 points across four benchmarks.

If you run agents in production, stop polishing the skill file and start compiling what you learned. The file ships. The notebook wins.

What follows is the memory map that makes it work, the notebook effect, the peeking failure, a promotion gate for skills that cross models, and a runnable miniature of the whole loop you can test in an hour.

How the notebook makes the coach better

WikiSkill splits the workspace into three layers with different jobs.

The raw/ folder keeps every execution trace and nobody ever edits it.

From those traces, the wiki/ folder compiles pattern pages, an index, a running log, and one append-only ledger called skill-impact.md.

And the skills/ folder holds the instruction files the working agent actually reads.

The loop runs four roles:

1. An actor agent attempts training tasks using only the current skills.
2. A maintainer agent then reads a sample of the traces, at most five failures and three successes, and patches the pattern pages.
3. A proposer agent studies the wiki index plus the full ledger, then digs through raw traces itself before suggesting exactly one skill change.
4. A strict gate keeps the change only if the validation score strictly improves. Rejected edits vanish from the skills folder and stay in the ledger forever.

The headline numbers for that switch: no wiki anywhere 48.7, proposer wiki 63.7, and 60.9 when the actor was allowed to peek.

The same switch also moved LiveMath 51.3 to 72.6 and SpreadSheet 49.9 to 76.6.

This asymmetric memory is the core design: the agent's active skill file instantly rolls back any failed edits to protect performance, but the coach's private wiki permanently records every failure so it never makes the same mistake twice.

On ALFWorld, a benchmark of text-world household errands, with Qwen-3.6-27B, the proposer's first idea was a vague skill named goal-directed-action, and the gate rejected it.

The ledger kept the diff and the rejection.

In the next round, the proposer wrote a concrete rule instead: never return an item to its origin location. This time it passed and got further refined by round four.

Instead of discarding failed attempts, the loop fed the rejected diff back to the coach as a negative example, turning a round-zero failure into the exact clue needed to ship a working rule in round one.

Most teams I've seen already keep the equivalent of the trace folder and none of the rest.

So if your loop has nowhere that remembers a rejected idea, you are paying full price to rediscover every dead end.