Tencent's Hy4 Opens a 770B Model With a 1M-Token Window
Tencent's new flagship mixture-of-experts model packs 770B parameters, native vision, and a 1M token context window under an open license.
Takeaways
- Tencent open-sourced Hy4, a 770B MoE with 49B active parameters and a 1M token context.
- Natively multimodal across text, image, and video via a 1.23B ViT feeding into the transformer stack.
- Uses Gated DeepSeek Sparse Attention with IndexCache plus identity Hyper-Connection residuals.
- Built-in DFly2 speculative decoder shares KV cache and sparse indices for constant-cost drafting.
- Ranks first among open-weight models on internal finance and research agent benchmarks.
- Available in BF16, FP8, and GGUF, with vLLM and SGLang support out of the box.
Tencent has released Hy4 weights, the open-weight flagship of its Hunyuan model family. The mixture-of-experts model has 770 billion total parameters, activates 49 billion per token, processes text, images, and video, and supports a 1,048,576-token context window.
The official repository provides BF16 and FP8 checkpoints alongside a system card and safety evaluations. A community GGUF build is also available for quantized deployments and experimentation.
770B total, 49B active
| Component | Specification |
|---|---|
| Language backbone | 78-layer mixture-of-experts transformer |
| Parameters | 770B total, 49B active per token |
| Expert routing | 256 routed experts and 1 shared expert per MoE layer; top 8 routed experts plus the shared expert activate for each token |
| Context window | 1,048,576 tokens |
| Vision encoder | 1.23B-parameter vision transformer with dynamic image and video resolution |
| Official formats | BF16 and FP8 |
| Serving frameworks | vLLM and SGLang |
The first transformer layer uses a dense feed-forward network, while the remaining 77 layers use MoE feed-forward networks. Activating 49B parameters lowers per-token computation, but deployment still requires access to the full checkpoint through GPU memory or slower offloading. Raw weight storage is roughly 1.54 TB in BF16 and 770 GB in FP8 before runtime overhead.
Sparse attention carries the 1M-token window
Hy4 controls long-context costs with Gated DeepSeek Sparse Attention and an IndexCache. An indexer selects the 2,048 positions considered most relevant to each token, while the cache reuses those selections across layers. This reduces attention work and index-selection overhead compared with scanning the entire context at every layer.
Identity Hyper-Connections widen the routes through which activations move between layers, expanding inter-layer information flow beyond a conventional residual pathway. The 1.23B vision transformer converts dynamic-resolution images and video frames into tokens that enter the same transformer stack as text.
Hy4 also includes DFly2, a speculative decoder designed with the sparse-attention system. Each draft layer shares its corresponding Hy4 layer’s key-value cache and selected positions, so draft attention examines approximately 2,048 keys per token regardless of the configured context length. Tencent says the drafter requires no additional KV-cache memory and transfers only one final-layer hidden state from prefill to decoding. Actual speedups will depend on draft-token acceptance rates, request lengths, kernels, and serving-framework support.
Tencent’s benchmark lead has limits
Tencent reports that Hy4 Base leads most knowledge and reasoning categories in its comparison with DeepSeek-V4-Pro-Base and DeepSeek-V4-flash-Base. The published results cited below come from Tencent’s model card and have not been independently reproduced.
| Benchmark | Area | Hy4 score | Reported comparison |
|---|---|---|---|
| MMLU | Knowledge | 91.6 | Leads the listed base-model comparison |
| MMLU-Pro | Knowledge and reasoning | 73.9 | Leads the listed base-model comparison |
| SuperGPQA | Graduate-level knowledge | 62.7 | Leads the listed base-model comparison |
| SimpleQA | Factual recall | 57.2 | Hy3 preview scored 26.4 |
| MMMU-Pro | Multimodal reasoning | 59.6 | Standard format with 10 options |
| CVBench | Visual understanding | 86.7 | No external comparator reported here |
| CountBenchQA | Visual counting | 90.6 | No external comparator reported here |
| MBPP-plus | Code generation | 74.0 | DeepSeek-V4-Pro-Base scored 79.1 |
Code generation is the clearest weakness in the selected public results, with Hy4 trailing DeepSeek-V4-Pro-Base by 5.1 points on MBPP-plus. Tencent reports multimodal scores only for the Hy4 family in the cited comparison, which limits conclusions about its position against external models.
Finance, games, and agents get targeted training
- Finance: Tencent evaluated 579 financial tasks with 213 finance professionals. Hy4 received an average score of 3.04 and ranked first among the open-weight models tested, ahead of DeepSeek-V4.1-Flash, Kimi K3.1 Pro, and GLM-5.3. The published summary does not provide enough detail about the scoring scale to interpret the absolute value.
- Hallucination reduction: Internal evaluations reduced the reported hallucination rate from 15.76% in the Hy4 preview to 5.45%. Tencent also reports that “speculative errors,” an internally defined category, fell from 23.1% to 5.6%.
- Game development: Training conducted with Tencent IEG developers targets Unreal Engine, Unity, and Godot workflows. Tencent reports gains of 19.1% on GameDev and 9.8% on GameLogic over the preview model.
- Computer use: Targeted Blender, CAD, command-line, and graphical-interface training focuses on planning across long, multi-application workflows.
- Research agents: Hy4 ranks first among the open-weight models Tencent tested on TenureTrail, an internal benchmark containing 123 research tasks across seven disciplines.
Eight GPUs for the FP8 path
Tencent officially supports deployment through vLLM and SGLang, with prebuilt Docker images tagged hy4. The company says the FP8 checkpoint can run across eight GPUs with tensor parallelism set to eight. That configuration assumes enough aggregate GPU memory, compatible FP8 hardware, and the required sparse-attention kernels.
docker run --gpus all --ipc=host -p 8000:8000 \
vllm/vllm-openai:hy4 tencent/Hy4-FP8 \
--tensor-parallel-size 8 \
--speculative-config '{"num_speculative_tokens":5,"method":"dfly"}' \
--attention-backend FLASHMLA_SPARSE \
--reasoning-parser hy_v4 \
--tool-call-parser hy_v4 \
--served-model-name hy4
The server exposes an OpenAI-compatible API. The runtime parameter reasoning_effort accepts "no_think" for direct responses and "high" for extended reasoning. Native tool calls are available when the hy_v4 reasoning and tool-call parsers are enabled.
A 1,048,576-token limit describes the maximum supported context, while practical latency and memory use still grow with prompt length. Sparse attention reduces the number of positions examined during attention, but long prompts still require tokenization, prefill computation, cache management, and data transfer.
Four caveats before adoption
- Frontier gaps remain: Tencent reports weaker performance against closed models on research-level mathematics, long-horizon coding, model self-improvement, and advanced visual question answering.
- Few-shot formatting can fail: Synthetic instruction data used during mid-training can cause the base model to miss expected answer formats, distorting evaluations that rely on exact matching.
- Full-window quality needs testing: The cited results do not establish retrieval accuracy, reasoning quality, or latency across the entire million-token window.
- The license is custom: Hy4 uses the Tencent Hy License, which is separate from standard OSI-approved licenses. Commercial deployment is governed by its specific use restrictions and obligations.
Why Hy4 changes the serving conversation
Recent Chinese open-weight flagships have largely come from DeepSeek, Moonshot’s Kimi, and Zhipu’s GLM. Hy4 adds Tencent to that group with a model that combines sparse million-token attention, cross-layer index reuse, multimodal inputs, and a KV-sharing speculative decoder in a deployable release.
The architecture also changes the assumptions behind serving infrastructure. Dense-attention stacks designed around 128K contexts cannot automatically realize Hy4’s advertised efficiency; deployments need compatible sparse kernels, DFly2 support, sufficient aggregate memory, and workload-specific tests of context retrieval, tool use, coding, and latency.