---
title: "LiquidAI's IFStruct Trains a 350M Model to Beat 4B Rivals at Structured Output"
subtitle: "Liquid AI's new open benchmark tests structured output compliance under realistic, messy prompts — and a 350M model trained on it beats models 10x its size"
url: "https://alphasignal.ai/news/liquidai-s-ifstruct-trains-a-350m-model-to-beat-4b-rivals-at-structured-output"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "Liquid AI"
company_url: "https://alphasignal.ai/company/liquidai"
published: "2026-06-30T14:02:26.000Z"
updated: "2026-06-30T15:03:00.714Z"
type: "News"
topics:
  - "Benchmarks"
  - "Llms"
subtopics:
  - "Structured Output"
  - "Small Models"
upvotes: 683
read_time: "5 min"
source_url: "https://huggingface.co/datasets/LiquidAI/ifstruct-v1.0"
comments: 0
---

# LiquidAI's IFStruct Trains a 350M Model to Beat 4B Rivals at Structured Output

> Liquid AI's new open benchmark tests structured output compliance under realistic, messy prompts — and a 350M model trained on it beats models 10x its size

Getting a model to reliably output valid JSON or YAML sounds like a solved problem. It is not. Structured output remains a common failure mode for language models, especially when schemas become complex and strings require careful escaping. Liquid AI just released [IFStruct](https://www.liquid.ai/blog/ifstruct-v1.0), an open benchmark designed to measure exactly this , and the results expose a gap that most existing evals quietly paper over.

## The problem with how we test structured output today

IFStruct targets something existing benchmarks miss: organic user requests present schema requirements in a variety of ways, often with additional constraints about formatting requirements. Most benchmarks hand the model a clean, finalized JSON Schema and ask it to comply. Real users don't do that. They paste an annotated example, describe the schema in plain English, switch formats halfway through, and add constraints like "no code fence" or "no commentary."

Constrained generation can enforce syntactic validity, but it cannot by itself make the model choose the right fields, values, or escaped content. Even under a schema constraint, the model's logits still need to meaningfully reflect the user's requested structure. IFStruct is scored without constrained decoding, so it measures whether the model actually understood the request , not whether the decoding machinery forced it into shape.

## What the benchmark actually tests

Each IFStruct prompt asks the model to generate a handful of instances of a randomized item (e.g. "Generate two recipes for blueberry pancakes"), with the schema requirements presented in a variety of different ways. The content itself is not assessed, only the structure. This isolation is intentional: the signal stays clean, without conflating schema-following with reasoning or data extraction ability.

The benchmark covers six distinct ways a user might specify a schema:

- A naturally written chat request ("hey can you whip up some invoices...")
- Bullet points with explicit field paths like `poems[].poem_text`
- Raw JSON Schema
- Annotated JSON or YAML structure examples
- Flat path glossaries
- ASCII field tables

About half of the prompts are rewritten by an LLM into humanlike prose, including "thinking-out-loud" requests that revise the spec before settling, to emulate organic requests rather than a neatly finalized schema. Many prompts also deliberately stress string escaping , embedded quotes, code snippets, file paths, and stack traces packed into otherwise-valid JSON or YAML, which is a common real-world breaking point.

Scoring is binary: a sample passes only if every constraint is satisfied. A sample passes only if all checks produce zero errors. Any error means a fail. The checks include parse validity, code-block requirements, commentary rules, item count, wrapper key names, field types, enum membership, numeric bounds, and nested list lengths.

## A 350M model that punches way above its weight

The task is highly learnable for small models: LFM2.5-350M, trained with RL on a dedicated held-out training split, can exceed the performance of far larger models like Qwen3.5-4B and granite-4.0-h-tiny. The numbers are striking.

| Model | IFStruct Score |
| --- | --- |
| gemma-4-31B-it | 95.90% |
| Qwen3-8B | 79.75% |
| granite-4.1-8b | 68.45% |
| **LFM2.5-350M + IFStruct RL** | **44.90%** |
| granite-4.0-h-tiny | 38.75% |
| Qwen3.5-4B | 36.25% |
| LFM2.5-350M (base) | 21.10% |

The base checkpoint passes just 21.10% of test samples. After RL, the same model scores 44.90%. That's more than double the baseline, achieved by training on a held-out split generated by the same pipeline that built the benchmark. The training method used is GRPO , a reinforcement learning technique (a variant of PPO) where the model learns from binary pass/fail rewards rather than from labeled examples.

## The flywheel: one generator, two uses

The real architectural insight here is that the benchmark and the training data share the same generator. The generative design of the dataset means easy production of training data. You don't need to hand-label anything. The same yes/no validator that scores the benchmark becomes the reward signal for RL training , a tight loop between evaluation and improvement.

This matters practically. If you want to fine-tune a small model for structured output in your own domain, you can:

1. Clone the [IFStruct repo](https://github.com/Liquid4All/ifstruct) and run the eval against your model to get a baseline
2. Generate a training split from the same pipeline
3. Train with GRPO using the binary validator as the reward function
4. Re-run the eval to measure improvement

The eval itself runs against any OpenAI-compatible endpoint. Setup is minimal:

```
git clone https://github.com/Liquid4All/ifstruct.git
cd ifstruct
uv sync
# then set BASE_URL and API_KEY in .env
uv run ifstruct-eval \
  --model your-model-name \
  --dataset data/test.jsonl \
  --results-file results/latest.json \
  --n-threads 64
```

## What it doesn't cover (yet)

The benchmark is deliberately narrow, and Liquid AI is upfront about the gaps. The design choice of assessing only the structural requirements, not content correctness or quality, means a respondent can ignore some of the content-related parts of the request and still pass, as long as they have followed the schema. If you train purely on IFStruct rewards, you'll want to add an LLM judge as a quality signal alongside it.

The difficulty of IFStruct requests is calibrated to be discriminative for low- to mid-ability models and saturates at the frontier ability level. Frontier models cluster near 100%, so it won't help you rank GPT-4o against Gemini 1.5 Pro. The validator also doesn't yet enforce regex patterns, date formats, email formats, or array uniqueness constraints , those are flagged as future work.

## Why this changes the calculus for small model deployment

The broader implication is that structured output compliance is a highly targeted, learnable skill , not a general capability that only large models can have. Frontier models can produce more or less perfect JSON and YAML to a schema, but small models still struggle, producing unparseable outputs and ignoring schema constraints. This is a very learnable task for even tiny (350M) models.

For teams running inference at the edge, on-device, or under strict cost constraints, IFStruct gives both a way to measure the problem and a recipe to fix it. The [dataset](https://huggingface.co/datasets/LiquidAI/ifstruct-v1.0) (2,000 prompts, Apache 2.0) is on Hugging Face now, and the benchmark code is open source.

## Sources

- [X post by @liquidai](https://x.com/liquidai/status/2071957538086506762)
- [IFStruct: Measuring structured-output compliance — Liquid AI Blog](https://www.liquid.ai/blog/ifstruct-v1.0)
- [Liquid4All/ifstruct — GitHub](https://github.com/Liquid4All/ifstruct)
- [LiquidAI/ifstruct-v1.0 — Hugging Face Datasets](https://huggingface.co/datasets/LiquidAI/ifstruct-v1.0)