Baidu Drops ERNIE 5.5 Flash, a Raw 280B Open Checkpoint Beating DeepSeek

Baidu has released ERNIE 5.5 Flash Base, a publicly downloadable sparse Mixture-of-Experts language model. The checkpoint captures the model immediately after pre-training, before specialized mid-training, supervised instruction tuning, or reinforcement learning from human feedback. Researchers and domain teams can apply their own post-training recipes without inheriting Baidu’s chat behavior or preference tuning.

A sparse MoE stores many feed-forward expert networks and routes each token through a small subset. ERNIE 5.5 Flash contains 280 billion parameters in total and activates 14 billion for each token, reducing per-token computation while retaining a much larger parameter pool.

The release in six figures

• Total parameters: 280B
• Parameters activated per token: 14B
• Pre-training data: 33.5T tokens
• Pre-training context: Up to 128K tokens
• Fine-tuning and inference context: Up to 1,048,576 tokens
• Weight formats: FP8 and BF16

The weights are available through the model card under a custom ERNIE 5.5 Flash Base license. The license governs use, modification, redistribution, and deployment, so teams should review its terms before adopting the model. Baidu’s release notes provide the architecture and evaluation details.

A base checkpoint by design

Raw pre-training checkpoints let researchers measure the effects of later training stages from a shared starting point. A team can compare supervised fine-tuning, reinforcement learning, domain-specific continued training, and alternative preference datasets without first undoing an existing instruction-tuning prior.

That flexibility also shifts work to the adopter. The model has no built-in chat alignment, assistant behavior, refusal policy, or tool-use format. Production deployments will generally require additional training, evaluation, prompt design, and safety controls.

Compression carries the long context

The network has 43 layers. One uses a dense feed-forward network, while 42 use Baidu’s Multi-Router Latent MoE design. Each MoE layer contains 512 routed experts, selects the top 10 for each token, and adds one shared expert.

The attention stack combines 37 Heavily Compressed Attention layers with six layers using Multi-head Latent Attention and dynamic sparse attention. This hybrid layout concentrates most attention work on compressed representations while preserving periodic access to finer token-level information.

Each HCA layer caches one entry for every 128 tokens, reducing its number of cached positions by roughly 128 times relative to token-by-token caching. The six token-level attention layers, wider HCA heads, and other runtime state make the model’s overall memory reduction smaller than that figure. Even so, the compressed cache makes very long inputs more practical.

The two attention types use different head widths. HCA layers operate over compressed sequences with 512-dimensional heads. The more compute-intensive MLA layers use 256-dimensional heads to limit the cost of token-level attention.

Several routers share the decision

A conventional MoE uses one routing network to rank experts for each token. Baidu says a single router’s scores correlate weakly with expert utility in deeper layers. Multi-Router Latent MoE combines several independently initialized sigmoid routers with different learned preferences, reducing dependence on one ranking signal.

The design preserves the top-10 expert count, nominal floating-point operations, and communication volume. Its intended gain comes from selecting a stronger combination of experts under the same activation budget.

ECHO turns the final layer into a draft model

Multi-token prediction trains a model to forecast several future tokens from the same position. ERNIE 5.5 Flash uses a method called ECHO, which reuses the backbone’s final layer for those additional predictions. The multi-horizon loss therefore updates parameters that also participate in standard next-token generation.

During speculative decoding, the same layer can propose draft tokens for verification by the full model. This removes the need to load and maintain a separate draft checkpoint. Draft generation and verification still consume compute, so the final speedup depends on proposal quality, hardware, and serving configuration.

Three supporting choices

• Manifold-constrained Hyper-Connections: An expansion rate of four replaces the conventional single residual stream with expanded, constrained connections.
• MultiMax output head: A learnable piecewise temperature adjusts the output distribution across different logit ranges.
• Numeric-aware tokenizer: A roughly 200K-entry BPE vocabulary groups numbers from right to left in chunks of up to three digits, preserving common numeric structure.

Baidu reports a broad benchmark lead

Baidu evaluated ERNIE 5.5 Flash against DeepSeek-V4-Flash-Base, Hy3-preview-Base, and MiMo-V2.5-Base under the same pipeline. Its model leads every benchmark in the published subset below. The largest margins over the strongest listed competitor are 9.2 points on MMLU-Pro and 6.6 points on LiveCodeBench.

The published results come from Baidu and have not been independently reproduced. Scores can change with prompting, sampling, tokenization, context settings, and evaluation harness versions, so comparisons remain most reliable when all models run through the same documented pipeline.

Baidu’s broader results also show ERNIE 5.5 Flash trailing at least one competitor on ChineseSimpleQA, INCLUDE-lite, HELMET-Rerank, and InfiniteBench. Those exceptions matter for deployments centered on factual retrieval, ranking, or sustained performance across very long inputs.

Repository data surfaces in the code scores

The pre-training corpus includes software-development records that extend beyond standalone source files:

• Commits, issues, pull requests, and code reviews
• Requirement-to-PR and multi-commit development trajectories
• Generated tests and fixes verified through continuous integration

That coverage is consistent with the model’s results on HumanEval+, LiveCodeBench, and RepoQA because those tasks reward execution feedback and repository-level context. The benchmark report does not isolate the contribution of this data from architecture, scale, tokenizer design, or the rest of the training mixture.

Serving requires multi-GPU infrastructure

Each token activates 14B parameters, yet a conventional deployment must make the full 280B parameter pool available. FP8 weights alone require roughly 280GB, while BF16 weights require roughly 560GB, before accounting for the KV cache, activations, communication buffers, and runtime overhead. Weight sharding or offloading can reduce memory per accelerator at the cost of additional communication or latency.

Baidu documents support for SGLang and vLLM. Its vLLM example pins version 0.32.0 and installs a dedicated plugin:

uv pip install vllm==0.32.0
uv pip install git+https://github.com/baidu/vLLM-ERNIE.git
vllm serve baidu/ERNIE-5.5-Flash-Base \
  --kv-cache-dtype fp8 \
  --enable-expert-parallel \
  --data-parallel-size 4 \
  --max-model-len 131072 \
  --moe-backend humming

The example caps requests at 131,072 tokens. Exposing the advertised 1,048,576-token limit requires a corresponding --max-model-len setting and enough memory for the larger cache and runtime state. Maximum accepted length describes serving support; retrieval quality and effective recall across the full window remain workload-dependent.

Applications should send prompts to the completions endpoint, commonly /v1/completions in an OpenAI-compatible server. The base checkpoint does not include the chat template or assistant tuning expected by a chat-completions API.

Who can use the raw checkpoint

Research groups can use the release to compare instruction tuning, reinforcement learning, data mixtures, and alignment methods from the same pre-training endpoint. Vertical-model teams gain direct control over domain behavior, output formats, refusal policies, and tool-use conventions, provided they have the compute and data required for post-training.

Architecture researchers can also study the components independently: compressed hybrid attention, multi-router expert selection, and a shared speculative-decoding head. For practitioners, the immediate constraints are equally concrete: a custom license, hundreds of gigabytes of weights, substantial serving infrastructure, and a checkpoint that still needs behavioral training before user-facing deployment.