Kaggle Lets Developers Build AI Benchmarks Locally With Claude Code

Kaggle just made writing AI evaluations feel a lot more like writing normal Python. The team shipped local development for Kaggle Benchmarks, which means you can now author, validate, push, and run benchmark tasks from VSCode, Cursor, Antigravity, or any agent shell instead of being trapped in Kaggle's hosted notebook editor.

Alongside the CLI, they released write-kaggle-benchmarks, an agent skill, essentially a structured instruction set you drop into Claude Code or a similar coding agent so it knows how to scaffold a task, run it against a model, and ship the results. The pitch is simple: describe the evaluation you want in natural language, let the agent generate the task file, and iterate locally.

Why this matters

Until now, creating evaluation tasks meant working exclusively in Kaggle's web-based notebook editor, instead of developers' preferred stack. The new update enables developers to create, validate, push, run and download tasks directly from their local development environments like Antigravity, VSCode, Cursor and coding agents.

That matters because benchmarks have become the bottleneck for evaluating frontier models. As AI models evolve from simple chatbots into reasoning agents that write code, use tools and solve complex problems, traditional benchmarks are no longer enough. The community needs dynamic, rigorous evaluations built by the people who use these models in the real-world. The global AI community has already created more than 10,000 evaluation tasks on the platform, and pulling that workflow into local tooling lowers the friction to add more.

What the SDK actually looks like

The library, kaggle-benchmarks, exposes a decorator-based API. It provides a structured framework for defining tasks, interacting with models, and asserting the correctness of their outputs. A minimal task is just a Python function:

import kaggle_benchmarks as kbench
@kbench.task(name="my-test-task")
def my_test_task(llm):
    response = llm.prompt("What is 2 + 2?")
    kbench.assertions.assert_in("4", response, expectation="Should contain 4")
my_test_task.run(kbench.llm)

The SDK goes well beyond string matching. You can go beyond simple string matching to test for code execution, tool use, and multi-turn conversational capabilities, and quickly test a model's capabilities on a new, creative task you've designed. Inputs can be structured dataclass or pydantic objects from models, and you can provide image, audio, and video inputs.

The CLI workflow

Everything routes through a new kaggle benchmarks command group (aliased to kaggle b). The documented loop is init, write, validate, push, run, status, download. Setup is one command:

kaggle b init -y

That fetches a short-lived Model Proxy API key, writes a .env, and drops an example_task.py plus a syntax reference next to it. The command fetches a short-lived Model Proxy API key and URL from Kaggle and appends them to your environment file. Because the key expires, you re-run kaggle b auth -y to refresh credentials without re-scaffolding.

Pushing and running a task against one or several models is straightforward:

kaggle b t push my-task -f task.py --wait
kaggle b t run my-task -m google/gemini-3.5-flash -m anthropic/claude-haiku-4-5
kaggle b t download my-task -o ./results

You can attach Kaggle datasets to the underlying notebook with repeated -d flags, stream live logs over SSE for running jobs, and publish a task to make it appear on a public leaderboard.

The agent skill, in plain terms

The interesting piece is the skill file itself. It consists of a series of structured instructions that guide a coding agent in building tasks utilizing the kaggle-benchmarks SDK and the Kaggle CLI. It teaches the agent the command hierarchy, the file format (jupytext percent cells with @kbench.task decorators), the LLM resolution precedence, and an explicit pacing rule that tells the agent not to chain the full pipeline. Each step is a checkpoint requiring user confirmation, which is a sensible guardrail given that pushing and running tasks burns real model proxy credits.

Sharp edges to know about

The skill file is unusually candid about silent failures. A few worth flagging:

• Missing .run() is a silent no-op. Push validation only checks for @task decorators, so a file without .run() executes on the server, produces no run file, and records nothing.
• The Model Proxy API key is short-lived. Auth errors mean re-running kaggle b auth -y before anything else.
• Multi-value flags must be repeated, not space-separated. Use -m a -m b, never -m a b.
• The CLI only manages tasks, not benchmarks. A benchmark is a curated collection of tasks; creating those collections still happens in the web UI.
• Delete is not wired up server-side yet and prints a not-supported message.

Where this slots in

For teams already running internal eval harnesses with tools like Inspect or lm-evaluation-harness, this is a path to publishing those evals against Kaggle's public leaderboards without rewriting them in a hosted notebook. For researchers, it means the loop of writing a probe, running it across a panel of frontier models, and downloading structured outputs all happens in your editor with your agent. The natural-language-to-eval pipeline is the real shift, since the bottleneck for community benchmarks has always been authoring effort, not compute.