Institute of Foundation Models Opens Full Training Stack Behind K2 Horizon 360
IFM has released K2 Horizon 360, bundling 380 checkpoints, 13T training tokens, RL infrastructure and a new attention architecture under Apache 2.0.
- IFM delivered K2 Horizon 360: 8 artifact categories and 30+ repos covering every LLM training stage, see artifact directory.
- 380 checkpoints across six models from pre-training through RL, letting you trace when capabilities emerged.
- 13 trillion pre-training tokens across 5 datasets, plus 11 data tool repositories.
- MoVA architecture applies MoE sparsity to attention value vectors with no extra KV-cache cost.
- RL360 ran 8,192 concurrent sandboxes, 150 steps in 7 days, ~1.3M rollouts for agentic RL.
- Uno diffusion LoRA delivers roughly 3x lossless decoding speedup on frozen autoregressive weights.
The artifact index from the Institute of Foundation Models now exposes the training machinery behind K2 Horizon, its six-model family released in September. K2 Horizon 360 spans eight artifact categories across more than 30 repositories, including pretraining data, intermediate checkpoints, training logs, infrastructure, reinforcement learning rollouts, and inference adapters.
The six-model fleet ranges from 0.9B to 375B parameters and carries an Apache 2.0 license. The models are available through Hugging Face, supported by vLLM and SGLang, and served by partners including Compass, Cerebras, AWS, and Nebius. Publishing the intermediate artifacts gives developers material for auditing, debugging, and studying the training process behind the final weights.
Inside the 360 Bundle
The release organizes the model lifecycle into eight categories:
- Checkpoints: 380 model snapshots across the six releases, covering pretraining through reinforcement learning. Each checkpoint captures the model’s learned parameters at a particular training stage.
- Data: 13 trillion pretraining tokens divided among five datasets and 24 subsets.
- Data tools: 11 repositories, including seven task-generation pipelines and three advanced-generation pipelines.
- Pretraining infrastructure:
xLLM, the production stack used to train the model family. - Post-training infrastructure:
RL360, a framework for long-horizon, agent-oriented reinforcement learning. - Training logs: 58 runs containing loss curves, gradient norms, and the configurations used for each run.
- Architecture:
MoVA, introduced in the 36B-A4B model. - Inference:
Uno, a diffusion-based adapter designed to accelerate decoding.
Among these artifacts, the 58 run logs provide a practical reference for teams training models in similar size classes. Loss curves show how quickly optimization progressed, while gradient norms help diagnose instability. Together with the saved configurations and checkpoints, they provide baselines for investigating divergence, regressions, and capability changes.
MoVA Makes Attention Sparse
Conventional mixture-of-experts models usually apply sparse routing in their feed-forward layers. Each token activates a small selection of expert multilayer perceptrons, while the attention layers remain dense. Mixture-of-Values, or MoVA, extends expert routing into the value projections used by attention.
MoVA replaces dense value projections with a learned pool of value experts and activates a subset for each token. The design remains compatible with FlashAttention, grouped-query attention, and sparse attention. IFM says the routing adds no extra key-value cache cost, which matters during long-context inference because that cache often dominates memory use.
The 36B-A4B checkpoint contains 100 routed experts, with eight active experts and one shared expert in each mixture-of-experts layer. Its attention layers contain 64 value experts, four of which activate per token. The model has 36 billion total parameters, uses roughly 4 billion per token, and supports a native 512,000-token context window. Under IFM’s matched training comparison, it scored slightly below the dense 32B baseline.
RL360 Separates Model Errors From Crashes
Long-horizon reinforcement learning introduces operational failures that can corrupt training signals. Agent rollouts may run for many steps, execute external tools, and depend on isolated sandboxes. A crashed environment can otherwise resemble a poor model decision when the system assigns rewards.
RL360 records infrastructure failures separately from model-generated errors. IFM reports that the framework operated 8,192 concurrent sandboxes, completed 150 training steps in seven days, and produced about 1.3 million rollouts. Those figures provide a concrete reference for the scheduling, isolation, and failure-handling demands of agent training at scale.
Combined with xLLM, which manages pretraining reconfiguration and recovery, RL360 covers two costly parts of large-model development: keeping distributed training jobs running and preserving reliable reward signals across large numbers of agent environments.
Uno Adds Parallel Decoding Through an Adapter
Uno uses diffusion distillation to generate blocks of tokens in parallel. The base Horizon model remains frozen, while a lightweight set of diffusion parameters learns the accelerated decoding process. IFM describes the resulting speedup as lossless because the autoregressive model continues to define the output distribution.
The diffusion parameters ship as a LoRA adapter, allowing teams to attach them to an existing deployment without retraining the full model. IFM reports roughly threefold decoding acceleration from the current adapters. Production adoption still calls for workload-specific measurements of latency, throughput, memory use, output consistency, and serving-framework compatibility.
Where Each Artifact Helps
| Goal | Relevant artifacts | Practical use |
|---|---|---|
| Reproduce or debug training | Checkpoints, logs, configurations, xLLM | Compare optimization behavior, locate divergence, and inspect intermediate model states. |
| Study capability emergence | 380 checkpoints | Run consistent evaluations across training stages to identify when measured capabilities appear. |
| Train agents | RL360 and rollout data | Examine long-running trajectories while separating environment failures from model errors. |
| Accelerate inference | Uno adapters | Test parallel decoding against existing latency, throughput, and quality targets. |
| Audit data composition | Datasets, subsets, and generation tools | Inspect dataset structure and the pipelines used to create training tasks. |
The artifacts improve observability across the training lifecycle, although exact reproduction still depends on matching hardware, software versions, numerical precision, distributed-training settings, and evaluation procedures. Capability-emergence studies also require stable prompts, scoring methods, and test sets across checkpoints.
Checks Before Committing Compute
Availability and independent validation remain uneven across the release. Some artifacts for the largest models were listed as staged during the September rollout, and third parties have yet to verify the flagship 375B-A23B model exhaustively. The surrounding ecosystem also has fewer integrations and deployment recipes than longer-established Llama and Qwen derivatives.
Developers should confirm each repository’s current files, license terms, dependencies, hardware requirements, and supported framework versions before planning a reproduction run. Even with those constraints, K2 Horizon 360 provides an unusually broad record of how a large model family moved from data preparation through pretraining, reinforcement learning, and deployment.
Comments
