Ant Group's E2LLM Squashes 400K Tokens Into a 7B Model Beating GPT-4o mini

Handling long context in language models has always been a game of tradeoffs. You can extend the position embeddings and swallow the quadratic attention cost, you can sparsify attention and lose track of information in the middle, or you can compress the prompt and hope you did not throw away the wrong tokens. Researchers from East China Normal University and Ant Group call this the impossible triangle, and they have built a system that sidesteps all three compromises at once.

Their framework, E2LLM (Encoder Elongated Large Language Models), pipes long inputs through a pretrained text encoder like BERT, squashes each chunk into a single embedding vector, and feeds those vectors into a decoder-only LLM as soft prompts. With Llama2-7B-chat as the decoder, the practical context window reaches roughly 400K tokens, and the theoretical maximum is the product of the encoder and decoder sequence lengths.

The impossible triangle problem

Long-context methods generally fall into three camps, and each one breaks a different constraint. Length extension methods like LongRoPE can push windows to 2M tokens but demand enormous continued pretraining with tens of billions of tokens and slow inference. Sparse attention drops the quadratic cost but ignores tokens that may hold key information, and it diverges from how the model was originally pretrained. Prompt compression, whether hard (RAG, LLMLingua) or soft (AutoCompressor), either introduces retrieval errors or forces the LLM to learn a new objective that fights its next-token-prediction training.

E2LLM's authors argue the three targets, strong long-context performance, low compute, and compatibility with pretrained weights, can only be reconciled by borrowing an idea from vision-language models: use a pretrained encoder to produce compact representations, then align them to the decoder with a lightweight adapter.