RELIA Toolkit Makes AI Release Decisions Replayable and Auditable
A trending open-source toolkit brings reproducible release gates, A/B ship decisions, and offline-verifiable records to real-time ML decision systems.
- Trending repo ai-decision-system packages release and A/B ship gates for real-time AI decision models.
- Three workflows: RELIA reliability gates, expgate A/B decisions, and a typed Jev decision model.
- RELIA enforces nine release gates on worst-of-five seeds, blocking any failure with reproducible synthetic benchmarks.
- Experiment gate emits SHIP, HOLD, ROLLBACK, or INVALID with CUPED and sequential monitoring support built in.
- Every decision produces an offline-verifiable JSON record with SHA-256 digest and replay-based verification CLI.
- Repository is MIT-licensed at root; RELIA itself remains an unreleased candidate pending Apache-2.0 clearance.
A GitHub toolkit makes AI release calls replayable
The GitHub project ai-decision-system has appeared on the platform’s trending charts with three Python workflows for recording and reviewing AI-related release decisions. Its two gating tools cover synthetic commerce models and live A/B tests, while a third workflow produces typed decisions from local models or the OpenAI Decisions API.
Its scope is release governance: separating evaluation data, measuring calibration, restricting model inputs, enforcing decision constraints, and preserving the evidence behind a ship, hold, or rollback call. These controls make approvals easier to reproduce and audit. Broader claims about safety, fairness, compliance, and commercial performance still require evidence from the deployment environment.
Three workflows, separate contracts
A workspace.json manifest treats each workflow as an independent project with its own tests and CI commands, as documented in the workspace guide.
| Workflow | Purpose | Runtime |
|---|---|---|
| RELIA | Evaluates release controls for synthetic commerce offers, retrieval, and ranking models. | Pinned Python 3.13 environment with a lockfile. |
| Experiment gate | Turns aggregate A/B test results into SHIP, HOLD, ROLLBACK, or INVALID. |
Python standard library only. |
| Jev decision model | Returns typed decisions from a local model or the OpenAI Decisions API. | Local or API-backed execution. |
The Jev workflow gives downstream code a stable set of typed fields and removes the need to parse free-form model responses. Its OpenAI-backed path requires network access and credentials, while local execution keeps generation on the machine.
RELIA tests release assumptions
RELIA is a reference implementation for models that make real-time commercial decisions, including offer selection and candidate ranking. It targets evaluation leakage, weak calibration, unauthorized feature access, decisions outside declared constraints, and regressions introduced by new data or retraining.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
