Skip to main content
Back

OpenAI's Harmony Fixes Broken Tool Calls in gpt-oss Local Deployments

OpenAI shipped a Rust-based renderer and parser for the token protocol that gpt-oss models require to reason, call tools, and stream structured output.

Pro

Repo, Full Access
  • OpenAI released harmony, a Rust renderer for the response format used by gpt-oss models.
  • The format separates output into analysis, commentary, and final channels for reasoning and tools.
  • Install with pip install openai-harmony, or add the Rust crate from GitHub directly.
  • Only needed for custom inference stacks; HuggingFace, Ollama, and vLLM handle it automatically.
  • Format mimics the OpenAI Responses API and enforces an instruction hierarchy across roles.
  • Rust core with pyo3 bindings delivers loss-free token round-tripping between render and parse.

If you have tried running gpt-oss locally and gotten garbled reasoning or broken tool calls, the culprit is almost always the prompt format. OpenAI released harmony, a Rust library with Python bindings that renders and parses the exact token protocol the gpt-oss open-weight models were trained on. Those models expect the harmony response format for conversation structure, reasoning output, and function calls, and without it they simply do not work correctly.

The repo has crossed 4.4k stars and sits at roughly 74% Rust, 26% Python. Encoding, parsing, and tokenization via tiktoken happen in Rust, with a thin pyo3 binding exposed to Python, so you get native-speed rendering from a normal pip install.

Harmony response format example showing system, developer, user, and assistant roles with channel-based output

A token protocol, not a template

Harmony is a structured token protocol, not a formatting convention you can bolt on after generation. It uses control tokens like <|start|>, <|end|>, <|message|>, and <|channel|> to segment conversations into roles and channels the model emits natively. The design mirrors the OpenAI Responses API, so anyone who has built against that surface will recognize the shape.