Korean Startup Motif Technologies Ships Motif 3, Beating Trillion-Parameter Giants on Agent Tasks
Motif Technologies released Motif 3, a 314B parameter sparse MoE with a novel GDLA attention, native 256K context, and MIT license.
PRO- Motif Technologies released Motif 3, a 314B-parameter MoE activating 13.2B per token, under MIT license.
- New attention design GDLA combines differential attention with MLA-style compressed KV cache.
- Pretrained on 12.5T tokens with MXFP8 expert GEMMs, Parallel Muon optimizer, and native 256K context.
- Post-training uses Multi-teacher On-Policy Distillation from seven RL specialists into one checkpoint.
- Leads open peers on Terminal-Bench 2.1 (74.9) and t3-Banking (35.3), lags on GPQA Diamond and SciCode.
- Ships with a vLLM fork, NVFP4 quant, and self-speculative decoding via built-in MTP head.
A relatively new player from South Korea just dropped one of the more architecturally interesting open-weight releases in recent memory. Motif 3 is a decoder-only mixture-of-experts model with 314 billion total parameters and 13.2 billion activated per token, shipped alongside a base checkpoint, a full technical report, and an MIT license. What makes it worth reading past the parameter count is that almost every major component, from attention to residual connections to activation functions, has been swapped out for something custom.
A wall of custom parts
Most frontier open models are variations on the same recipe: grouped-query attention, SwiGLU, RMSNorm, standard residuals. Motif 3 changes almost all of those. The headline swap is Grouped Differential Latent Attention (GDLA), which integrates grouped differential attention (asymmetric signal/noise heads with a token-dependent differential coefficient) with the compressed KV latent of Multi-head Latent Attention, plus a query-dependent output gate. In plain terms, it borrows the trick from Differential Transformer of subtracting a noise attention map from a signal attention map to sharpen focus, then compresses the KV cache the way DeepSeek's MLA does to keep memory manageable.
The rest of the block is unusual too:
- Expert-Specific PolyNorm: replaces the SiLU gate with a learned polynomial normalization whose coefficients are learned independently per expert, reducing activation outliers while allowing each expert to specialize.
- Modified manifold-constrained hyper-connections: instead of a plain residual add, doubly-stochastic mixing of 4 parallel residual streams, with the post-mapping multiplier annealed from 2 to 1 during pretraining. The goal is to stop activation magnitudes from blowing up as depth grows.
- Multi-Token Prediction head: a 1-layer MTP head enabling self-speculative decoding at inference, so the model can draft its own next tokens without a separate draft model.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
