Japan's NII Releases LLM-jp-4 to Beat GPT-4o on Japanese Benchmarks
Japan's National Institute of Informatics releases an 8B dense and 32B MoE model trained from scratch on 12T tokens, with the full pipeline open to inspection.
- NII released LLM-jp-4, an 8B dense and 32B-A3B MoE model trained from scratch on ~12T tokens.
- 32B-A3B scored 7.82 on Japanese MT-Bench, beating GPT-4o's 7.29 under a GPT-5.4 judge.
- MoE uses 128 experts with 8 active per token, giving 3.8B active parameters out of 32.1B total.
- Both models support 65,536-token context and ship under Apache 2.0 for commercial use.
- Full training scripts, corpus composition, and selection criteria are published for reproducibility.
- Three-stage pipeline: pre-training, mid-training with QA and context extension, SFT+DPO for thinking mode.
NII releases LLM-jp-4, a from-scratch Japanese model family
Japan’s National Institute of Informatics (NII) has released LLM-jp-4, a family of Japanese large language models designed for transparent, reproducible deployment. It includes an 8-billion-parameter dense model and a mixture-of-experts model with 32.1 billion total parameters and 3.8 billion active per token. The paper describes roughly 11.7 trillion training tokens for each model.
The authors report a Japanese MT-Bench score of 7.82 for the 32B-A3B model, using GPT-5.4 as the automated evaluator. Their results place it above GPT-4o on that evaluation. The release gives organizations handling Japanese data an Apache-licensed option whose weights, training process, and infrastructure are governed domestically.
From-scratch weights, auditable inputs
Fine-tuning Llama, Qwen, Mistral, or another existing model carries forward its base weights, license, and partially documented training history. LLM-jp-4 removes that dependency by training new weights from an assembled corpus while retaining established architectural ideas.
The dense model uses a Llama-style architecture, while the sparse model adopts the Qwen3MoE design. Neither model is a fine-tune or distillation of existing base weights.
NII documents its criteria for selecting data, external models, and software. It also publishes pretraining code, including scripts for the 8B model and the MoE model.
The weights use the Apache 2.0 license, which permits commercial use, redistribution, and modification subject to its terms. Dataset licenses, privacy obligations, output risks, and local regulations still require separate review before production deployment.
A three-stage route to 64K
Both models pass through pretraining, mid-training, and post-training, with each stage changing the data mix or model behavior.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
