Tongyi Lab's HydraHead Beats Long-Context Rivals Trained on 15B Tokens

Tongyi Lab's latest weekly roundup packages three threads worth paying attention to: a fresh long-context attention design called HydraHead, community quantization work that squeezes Qwen3.6-27B onto a single 24GB card, and the first installment of a series where researchers narrate the thinking behind their papers, starting with the scientific foundation model LOGOS.

Rethinking hybrid attention at the head level

The headline research piece is HydraHead, a hybrid attention architecture that mixes Full Attention (FA) and Linear Attention (LA) inside individual layers rather than assigning each layer one or the other. The quadratic complexity of attention poses a critical bottleneck for long-context processing, and most open-source hybrid models adopt a layer-wise strategy, though prior work has noted the inherent difficulty of integrating Linear Attention with Full Attention.

The team's motivation came from an interpretability study of how heads behave. Layers exhibit block-wise functional similarity, while individual heads within the same layer display distinct functional specialization despite sharing input features, suggesting that the head dimension provides a natural and principled granularity for fusing heterogeneous attention signals. Put more simply: neighboring layers do mostly the same thing, but heads within a layer specialize, so the head is the right knob to turn.

HydraHead's recipe has two moving parts:

• An interpretability-driven strategy that preserves Full Attention only for retrieval-critical heads
• A scale-normalized fusion module that enables Full and Linear Attention to coexist within the same layer

The efficiency numbers are what make it interesting. With interpretability-driven head selection, it matches a 3:1 layer-wise hybrid's long-context performance at a 7:1 LA-to-FA ratio, and trained on only 15B tokens, HydraHead achieves over 69% improvement over the baseline at 512K context length, approaching Qwen3.5, a leading model of comparable size with a native context length of 256K. That is a lot of long-context capability from a very small training budget, which matters if you are trying to retrofit hybrid attention into an existing checkpoint without a full pretraining run.

The paper also leans on a three-stage transfer pipeline with parameter reuse and distillation to convert dense FA models into HydraHead hybrids cheaply, which is likely the more practically useful contribution for teams already sitting on trained weights.

Qwen3.6-27B, now cheaper to run

The second thread is community tooling. Unsloth's quantization of Qwen3.6-27B has been updated with new formats aimed at both consumer and datacenter hardware. Their NVFP4 checkpoint delivers 2.5x faster throughput than other NVFP4 quants, is calibrated on a mixture of the Unsloth dataset plus UltraChat, and works on a 24GB VRAM GPU.

A few practical notes from Unsloth's guide:

• The new dynamic NVFP4 Qwen3.6 quants run around 2.5x faster than other NVFP4 quants, letting Qwen3.6-27B run on 24GB VRAM and Qwen3.6-35B-A3B 1.7x faster on 32GB VRAM.
• NVFP4 requires NVIDIA's Blackwell GPUs like the RTX 50 series, DGX Spark, B200, and B300. For older cards, GGUFs remain the path.
• Qwen3.6-27B can now hit 140 tokens per second generation with UD-Q2_K_XL, and Qwen3.6-35B-A3B reaches 220 tokens per second.
• Context length is 262,144 natively and extensible up to 1,010,000 tokens.

If you want to launch a server quickly, the recommended path uses SGLang:

python -m sglang.launch_server \
  --model-path Qwen/Qwen3.6-27B \
  --port 8000 --tp-size 8 \
  --mem-fraction-static 0.8 \
  --context-length 262144 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder

A common language for science

The third piece is the debut of Ready to Share, a series where Tongyi researchers walk through the reasoning behind their work. The series invites researchers from the team to discuss the ideas behind their latest work, not just the final results, but also the questions, experiments, and decisions that shaped the research.

Episode one covers LOGOS, a foundation model trying to represent proteins, molecules, materials, and chemical reactions in one shared grammar. The results are broader than a single benchmark win: LOGOS achieved competitive accuracy on benchmark datasets and identified additional candidate pockets that have not yet been experimentally verified, and the team is looking for collaborators to validate these predictions through wet-lab experiments.

It also generalizes. On materials generation, specifically metal-organic frameworks (MOFs), LOGOS again achieved state-of-the-art performance, and on reverse reactions in chemistry, deducing possible combinations of reactants from a product, it also worked well, suggesting the unified scientific grammar generalizes across multiple scientific domains. To test whether the model behaves like an LLM, they trained models ranging from 0.6B to 8B parameters, which suggests that scaling laws may also apply to scientific foundation models. Weights and checkpoints live on the LOGOS-Hub organization on Hugging Face.

Why the bundle matters

Read together, the three items point in the same direction. HydraHead attacks the cost of long context at the architecture level. The Unsloth quants attack it at the deployment level, making a 262K-context model runnable on a single consumer GPU. LOGOS pushes on the applications side, testing whether the same recipes that scaled LLMs generalize to molecules and materials. None of these individually would be a headline release, but the combination is a useful snapshot of where an open-source lab is spending its research budget: efficiency of attention, accessibility of inference, and expansion into scientific domains.