Meta's WildArtifactBench Ditches Fixed Rubrics to Judge Real-World Agents

Benchmarks tend to lie about how good agents actually are. Meta is trying to fix that with WildArtifactBench, an internal evaluation framework it uses on its Muse Spark models, and it just released 10 sample tasks so outsiders can poke at the methodology.

The pitch is straightforward: strong performance on public benchmarks shows one side of the story, but frontier models today still struggle to execute real-world agentic tasks such as making production-level games, and Meta wants to evaluate agents across a wide spectrum of multimodal tasks with practical utility. WildArtifactBench is the mechanism.

Ditching ground-truth rubrics for preference votes

Most benchmarks require a single correct answer or a deterministic rubric. That constraint is why they cluster around math, code contests, and multiple choice questions. WildArtifactBench flips the setup.

Meta describes it as an evaluation used to score agents on any task, with any deliverable format, and any level of verifiability. Instead of grading against a fixed answer key, it pits two agents against each other and asks a judge which output is better. The scoring system that falls out of this looks like:

• Win rates from head-to-head comparisons between a tested agent and a baseline
• Elo scores derived from those matchups, similar to how chess engines are ranked
• Preference votes from both human raters and other agentic judges

The upside, according to Meta, is that win rate and Elo scores are obtained by comparing the deliverables of the tested and baseline agents, with both agentic judges and human judges able to do the comparison and choose a preference, which greatly broadens task coverage by removing the need for ground-truth deliverables and objective rubrics.

What the 10 preview tasks actually look like

The preview drop covers a genuinely weird spread of deliverables. Categories include Audio Analysis, Medical and Healthcare, Digital Design, Machine Learning Engineering, Engineering/Materials, 3D Modeling, and Video Editing. Concrete examples:

• Plywood Whale: produce an SVG of interlocking cutout pieces for a 3mm laser-cut plywood desk ornament, with tabs and slots that actually fit
• Cat Mesh: a 3D digital design task
• DQN Driving: build a deep Q-network driving trainer as a machine learning engineering task
• Cardiac Ultrasound and Coronary Stenosis: medical video and image analysis
• Bird Sound: audio analysis with 47 input files
• Materials Metrology: quantitative analysis of engineering imagery

Each task ships with an instruction.md, source data/, and a rubrics.json. The Plywood Whale task, for instance, defines 11 criteria ranging from whether the SVG is valid to whether the tabs and slots are geometrically consistent so pieces actually mate together, capped by a rule that assemblability is prioritized over visual complexity, so a simple design that can actually be assembled is strongly preferred over a detailed or beautiful design whose pieces do not fit together. That is the kind of judgment call that break traditional autograders.

Why this matters for agent evaluation

The framework is showing up alongside Meta's push on multimodal agents. Muse Spark 1.2 combines deep video understanding and dense captioning with a powerful suite of agentic tools, including web development, real-time search, and spatial grounding, so the model can move beyond passive observation toward autonomous orchestration of complex workflows that translate audio-visual input into actionable solutions. You cannot score any of that with a multiple choice test.

WildArtifactBench also tracks cost, plotting automatic and human evaluation scores against average cost per task on a Pareto frontier. That matters if you actually deploy agents: an eval that reports quality alone can hide the fact that a model is winning by burning ten times the tokens of its baseline.

The catch

Preference-based evaluation is not free of problems. LLM judges have well-documented biases toward longer, more confidently written outputs, and toward responses that resemble their own style. Elo systems also drift with the baseline you pick, and with only 10 preview tasks and a handful of inputs per task (several tasks list a single input file), the sample is thin. Meta is asking you to trust the methodology more than the numbers at this stage.

Still, if you are building agents that produce SVGs, 3D meshes, edited video, or ML training code, the preview tasks are worth grabbing. They are unusually close to the kind of deliverable a real user would ask for, complete with rubric files you can reuse in your own eval pipeline. Even if you never touch Muse Spark, having a shared vocabulary for judging agent artifacts across weird domains is the useful part.