AutoTrust's AutoTrust-JEV Turns AI Decisions Into a Single Token at 97% Accuracy
AutoTrust's 27B decision model outputs calibrated option probabilities instead of free text, topping tool selection and intent classification benchmarks.
- AutoTrust-JEV-27B v1.1 is a 27B text decision model returning calibrated option probabilities instead of free text.
- Scores 67.31 on the Decision Index 0.3 suite across 37 benchmarks.
- Tools and Automation leads at 85.19, with BFCL 96.25 and CLINC150+OOS 97.06.
- Weak on HLE, ChessBench, GPQA Diamond and other open-ended reasoning tasks.
- Apache-2.0 licensed, served via vLLM with a bidirectional prefix plugin on one 80GB GPU.
- Best fit for tool selection, intent routing, claim verification and closed-option NLI tasks.
AutoTrust-JEV-27B v1.1, a 27-billion-parameter text decision model, has reached Hugging Face’s trending list. It accepts a question with a finite set of answers, emits one token restricted to the option codes, and converts the token log-probabilities into a distribution over those choices. The model card reports a 67.31 score on Decision Index 0.3, with its strongest results in tool selection, intent routing, and automation.
One token, finite choices
Classification with a general-purpose language model usually involves generating text and parsing the response into an expected label or schema. AutoTrust-JEV uses a shipped decision engine to format each question, call a vLLM backend, constrain the next token to valid option codes, and convert the returned log-probabilities into option-level scores. It supports multiple-choice and yes-or-no questions with text-only inputs.
Restricting the output removes malformed JSON, unexpected prose, and labels outside the supplied option set. The resulting probabilities can support ranking, thresholding, and routing, although they still require calibration on representative application data. Production systems should set thresholds on held-out data and provide an abstention or escalation path for low-confidence cases.
Routing and tool use lead the scores
The model card reports a public Decision Index of 67.31 across 37 indexed benchmarks and 140,178 completed, scoreable requests. Tools and Automation is the highest-scoring of the suite’s five areas at 85.19 skill points.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.