Epoch's FAB Benchmark Watched GPT-6 Astra Jump from 28% to 80%

Epoch AI’s Furniture Assembly Benchmark (FAB) tests whether multimodal models can compare an IKEA manual with a photo of an in-progress build, determine whether the assembly is correct, and locate any mistake. The leading score rose from 28% to 80% in ten months, while the fastest model reduced median analysis time to three minutes. Those gains point toward practical visual assistants for inspection, repair, and guided assembly.

A blind spot in vision evaluations

Most vision evaluations focus on image captioning, visual question answering (VQA), or document parsing. Those tasks do not measure whether a model can inspect a photograph of a physical object and compare its spatial details with step-by-step diagrams.

Epoch treats furniture assembly as a narrow proxy for related work such as vehicle maintenance and appliance repair. The shared challenge is identifying parts, orientation, placement, and assembly order from imperfect real-world images.

Sixty photos, three builds

Epoch selected three IKEA products with increasing scores on the IKEA Complexity Index, which multiplies the number of assembly steps by the number of pieces.

Researchers photographed each build at multiple stages and deliberately introduced realistic errors. They sometimes continued assembling after a mistake, obscuring its origin behind later work. The finished benchmark contains 60 images split between correct and flawed builds.

During each evaluation, the model receives a photograph, the relevant instruction PDF, a tool for zooming into the image at higher resolution, and a Python interpreter. It works inside an agentic sandbox and can take up to 80 steps to inspect the materials and produce an answer.

Scoring handles the two image classes differently:

• Flawed builds: The model must identify the incorrect assembly step and describe the mistake accurately.
• Correct builds: The model must report that no assembly error is visible.

An LLM judge, GPT-5.6 Sol, decides whether each error description qualifies for credit. Epoch instructed the judge to grade leniently.

Ten months, 52 points

Claude Opus 4.5 led the benchmark with 28% in November 2025. Over the following ten months, GPT-6 Astra raised the leading score to 80%.

Assembly complexity did not reliably predict model performance. Epoch found that error subtlety, camera angle, and the amount of work completed after a mistake appeared to matter more than the number of parts.

Failure costs shape model choice

Google’s Gemini and Alibaba’s Qwen models usually predict that an error exists, producing many false positives. Early Anthropic and OpenAI models tend to overlook real mistakes, producing false negatives. Gemini 3.1 Pro correctly accepts almost no error-free builds. GPT-5.4 detects almost no flawed ones.

• Sensitivity measures how often a system catches real errors.
• Specificity measures how often it accepts correct builds.

Safety-critical inspection systems generally prioritize sensitivity because a missed defect can cause damage or injury. Consumer assistants also need enough specificity to avoid repeatedly asking users to dismantle correct work. Aggregate benchmark scores can conceal that operational difference.

Three minutes per inspection

GPT-6 Astra combines the highest score with a median processing time of three minutes per photo. It is the fastest evaluated model and runs between two and ten times as quickly as the previous leaders.

Epoch considers three minutes too slow for fluid, real-time guidance. The result still narrows the latency gap for agents that must inspect each stage of a repair or assembly task before a user proceeds.

Open weights remain behind

Open-weight models, whose parameters are available for developers to run and adapt, trail closed systems more sharply on FAB than on broader capability tests. Epoch estimates that Kimi K3, the highest-scoring Chinese model tested, lagged the frontier by seven months when released. Its measured lag on the Epoch Capabilities Index was 4.4 months.

Model availability also limits the field. DeepSeek V4 Pro, DeepSeek V4 Flash, and Z.ai’s GLM 5.3 lack image support, leaving few open-weight options that combine vision with tool use.

Validate beyond the leaderboard

FAB contains 60 images from three products made by one manufacturer. Its staged errors and controlled photographs cannot establish how models will perform across different tools, lighting conditions, camera angles, product categories, or user-generated images. The lenient LLM judge introduces another source of measurement uncertainty.

Teams building visual inspection or repair systems can use FAB to shortlist models, then test those candidates against their own image conditions and the relative costs of false alarms and missed defects. Epoch’s full results include an interactive leaderboard, per-item heatmaps, and a chart comparing accuracy with processing time.