---
title: "HUST's MoDA Replaces a Decade-Old AI Building Block With 2.11% Accuracy Gains"
subtitle: "HUST's hustvl group open-sources MoDA, replacing residual connections with attention across transformer layers for a 2.11% accuracy gain at only 3.7% extra compute."
url: "https://alphasignal.ai/news/hust-s-moda-replaces-a-decade-old-ai-building-block-with-2-11-accuracy-gains"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-04-21T05:22:53.000Z"
updated: "2026-08-18T06:28:57.674Z"
type: "Repo"
topics:
  - "Llms"
  - "Training Infra"
subtopics:
  - "Scaling Laws"
upvotes: 1933
source_url: "https://github.com/hustvl/MoDA"
comments: 0
truncated: true
---

# HUST's MoDA Replaces a Decade-Old AI Building Block With 2.11% Accuracy Gains

> HUST's hustvl group open-sources MoDA, replacing residual connections with attention across transformer layers for a 2.11% accuracy gain at only 3.7% extra compute.

Since ResNet in 2015, every deep learning model has communicated between layers the same way: add the output of one layer to the next, `x + F(x)`. It is so fundamental that most practitioners never question it. The [hustvl research group at Huazhong University of Science and Technology (HUST)](https://github.com/hustvl/MoDA) just open-sourced **MoDA (Mixture-of-Depths Attention)**, a hardware-efficient implementation of a mechanism that replaces that additive residual with attention across the depth dimension of a transformer, letting any layer directly query the key-value pairs of every layer that came before it.

## The problem with depth

Scaling depth is a key driver for large language models, yet as LLMs become deeper they often suffer from signal degradation: informative features formed in shallow layers are gradually diluted by repeated residual updates, making them harder to recover in deeper layers. Think of it as a game of telephone where every person in the chain also hears every prior message simultaneously. By person 152, the original message from person 3 is technically present but buried in a chorus of 151 accumulated additions.

Prior work tried to fix this by blending layer outputs more cleverly. DenseNet kept every layer's output. DenseFormer and LIMe learned static mixing weights. Hyper-Connections widened the residual to multiple channels. But all of these approaches share the same conceptual frame: **accumulation**. They decide how much to draw from each earlier layer based on the current layer's state alone, without consulting what those earlier layers actually contain. The HUST team argues this is a category error. Inter-layer communication should be **retrieval**, not accumulation.

## Attention in two dimensions at once

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [hustvl/MoDA GitHub Repository](https://github.com/hustvl/MoDA)
- [Mixture-of-Depths Attention — arXiv 2603.15619](https://arxiv.org/abs/2603.15619)
- [The Second Half of Model Architecture — Lianghui Zhu](https://lh-zhu.github.io/The-Second-Half-of-Model-Architecture/)
- [Mixture-of-Depths Attention — Hugging Face Papers](https://huggingface.co/papers/2603.15619)