OTel 2.0 Turns Google's Gemma 4 Into a One-Token Decision Model

Farbod Tavakkoli has published OTel-2.0-Decision-26B-A4B-4bit, a 4-bit quantized derivative of Google’s Gemma 4 26B mixture-of-experts instruction model. The checkpoint exposes Gemma through a fixed-label decision interface for classification, yes-or-no questions, and ordinal scoring.

The release matters for systems that need constrained outputs and usable confidence signals. Each request generates one token, while the application reads the probabilities assigned to labels such as A, B, and C. That removes free-form response parsing and supports thresholds, abstention, and routing through an OpenAI-compatible API.

• Inputs: messages, tickets, transcripts, logs, or documents paired with a question and fixed labels.
• Output: one label token plus its probability and the total probability assigned to valid labels.
• Serving: vLLM 0.30.0, with documented ROCm paths for AMD Instinct GPUs.
• Excluded workloads: open-ended chat, retrieval, document ranking, and long-form generation.

A chat endpoint for fixed labels

Developers send the source text, a question, and a compact answer set through the chat-completions API. The prompt should require a single letter, and every label should map to one tokenizer token.

question = """State:
Order #1182. Status: shipped on 3 September. Carrier: DHL.

Question: Has the order been shipped?

Options:
A. Yes
B. No

Answer with the letter only."""

r = client.chat.completions.create(
    model="OTel-2.0-Decision-26B-A4B-4bit",
    temperature=0,
    max_tokens=1,
    logprobs=True,
    top_logprobs=64,
    messages=[{"role": "user", "content": question}],
)

The first generated position contains the candidate probabilities. A minimal extraction path looks like this:

from math import exp

labels = {"A", "B"}
top = r.choices[0].logprobs.content[0].top_logprobs

label_probs = {label: 0.0 for label in labels}
seen = set()

for item in top:
    label = item.token.strip()
    if label in labels:
        label_probs[label] += exp(item.logprob)
        seen.add(label)

if seen != labels:
    raise RuntimeError(
        "Some labels were omitted from top_logprobs; "
        "increase the limit or reject the result."
    )

decision = max(label_probs, key=label_probs.get)
answer_mass = sum(label_probs.values())