---
title: "Nous Research's Lighthouse Attention Trains 1.69x Faster on Million-Token Contexts"
subtitle: "Nous Research's Lighthouse Attention cuts long-context LLM pre-training time by 1.4–1.7x, running 17x faster than standard attention at 512K context — with no custom sparse kernel and no quality loss at inference."
url: "https://alphasignal.ai/news/nous-research-s-lighthouse-attention-trains-1-69x-faster-on-million-token"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "Nous Research"
company_url: "https://alphasignal.ai/company/nousresearch"
published: "2026-05-07T16:49:28.000Z"
updated: "2026-06-09T07:25:31.264Z"
type: "Paper"
topics:
  - "Llms"
  - "Training Infra"
subtopics:
  - "Long Context"
  - "Pretraining"
upvotes: 1024
source_url: "https://arxiv.org/abs/2605.06554"
comments: 0
truncated: true
---

# Nous Research's Lighthouse Attention Trains 1.69x Faster on Million-Token Contexts

> Nous Research's Lighthouse Attention cuts long-context LLM pre-training time by 1.4–1.7x, running 17x faster than standard attention at 512K context — with no custom sparse kernel and no quality loss at inference.

Training a transformer on sequences longer than 100K tokens is expensive in a very specific way: attention cost grows quadratically with sequence length. Double the context, quadruple the compute. [Lighthouse Attention](https://arxiv.org/abs/2605.06554), released by Nous Research, attacks this problem at the training stage with a hierarchical selection mechanism that wraps around standard attention , and can be completely removed before inference, leaving behind a normal dense model.

## The wall that FlashAttention can't move

Scaled dot-product attention (SDPA) has O(N²) time and memory complexity. FlashAttention reduces constants but not asymptotics; at sequence lengths above 100K, the attention term dominates all other compute in the model. The frontier of language modeling has moved to 128K, 1M, and longer contexts , pushed by agentic reasoning, long-document understanding, and multimodal inputs , making this bottleneck increasingly costly.

Prior work tried to fix this with sparse attention: instead of attending to every token, each query attends to a small subset. Methods like MoBA and Native Sparse Attention (NSA) select contiguous blocks; DeepSeek Sparse Attention (DSA) uses a learned indexer to pick the top-k tokens. These methods produce meaningful inference speedups but inherit two design decisions that fit long-context pretraining poorly: asymmetry (queries stay at full resolution while keys and values are pooled) and architectural entanglement (selection lives inside the attention kernel, so carefully optimized dense-attention kernels cannot be reused).

## A pyramid that scores itself

Lighthouse replaces a standard attention layer with a four-stage pipeline that surrounds, but does not modify, the attention kernel: a pre-attention selection stage drives a contiguous gather, stock FlashAttention runs on the gathered sub-sequence, and a post-attention scatter writes the result back to the original positions. The key insight is that the selection logic is entirely decoupled from the attention kernel itself.

![Needle-in-a-Haystack retrieval heatmaps comparing Lighthouse variants against the dense SDPA baseline across context lengths up to 96K](https://arxiv.org/html/2605.06554v1/figures/niah_combined.png) 

Here's how the four stages work:

1. **Pyramid Pool:** Q, K, and V are all average-pooled symmetrically into an L-level pyramid with pooling factor p. Each level summarizes p times more tokens than the level below it. Unlike all prior hierarchical sparse methods (NSA, HISA, InfLLM-V2), which only pool the key-value side, Lighthouse pools queries too.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Long Context Pre-Training with Lighthouse Attention (arXiv)](https://arxiv.org/abs/2605.06554)
- [Lighthouse Attention GitHub Repository](https://github.com/ighoshsubho/lighthouse-attention)
- [Nous Research Tweet announcing Lighthouse Attention](https://twitter.com/NousResearch/status/2055337939270332862)