Laya Runs AI Decisions in 7ms on Apple Silicon Without Cloud APIs
A native MLX port of the Laya typed decision model runs 7-14 ms structured classifications locally on Apple Silicon, no text generation required.
- Native MLX port of the Laya typed decision model for Apple Silicon.
- 13.4 ms median latency for short English decisions on M3 Max, 7.4 ms multilingual.
- Peaks under 1 GiB memory; 395 questions per second batched throughput on multilingual checkpoint.
- Bidirectional encoder with choice, score, and P(true) decision heads instead of text generation.
- Pre-converted FP16 weights on Hugging Face, install with pip install laya-mlx.
- Matches upstream PyTorch on 63/63 validation questions; Apache-2.0 licensed independent port.
laya-mlx brings typed decision models to Apple Silicon
laya-mlx ports the Laya family of typed decision models to Apple’s MLX framework. The Python package runs pretrained FP16 checkpoints locally on Apple Silicon, with project benchmarks reporting median single-query latency between 7.39 and 13.42 ms on an M3 Max.
Applications define a choice, score, or proposition in advance, and dedicated decision heads return calibrated probabilities. This avoids token-by-token decoding, generated JSON, and the associated parsing path. Model inference stays on the Mac and requires no PyTorch runtime or cloud API.
Milliseconds, with conditions
The project measured end-to-end latency on an M3 Max with a 40-core GPU and 128 GiB of unified memory. Each single-query result covers one short English question; throughput figures use batches of 50. These are project-reported measurements and await independent reproduction.
| Checkpoint | Parameters | Single-query median | Batch-50 throughput |
|---|---|---|---|
| Laya | 421M | 13.42 ms | 146.8 questions/s |
| Multilingual Laya | 322M | 7.39 ms | 395.0 questions/s |
Both benchmark configurations kept reported peak memory below 1 GiB. Batch throughput measures aggregate processing and should not be read as per-request latency.
The included Snake demo runs the model locally at approximately 60 decisions per second and displays per-move probabilities. A separate cycle-safety layer can reject unsafe proposals before the game applies them.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
