Cohere Labs' Aya-23-8B Beats Mistral and Gemma Across 23 Languages
Cohere Labs' Aya-23-8B brings instruction-following quality to 23 languages, beating Mistral and Gemma in multilingual benchmarks with an open-weights 8B model
PRO- Aya-23-8B is an open-weights 8B instruction-tuned model from Cohere Labs supporting 23 languages, available on Hugging Face.
- Beats Mistral-7B and Gemma-1.1-7B in multilingual benchmarks, with GPT-4-judged win rates of 65%+ across 10 languages.
- Built on Cohere's Command R base with a 256k multilingual BPE tokenizer, GQA, SwiGLU, and RoPE; trained on ~10.5M multilingual instruction samples.
- Strongest for translation, summarization, and open-ended generation in non-European languages like Arabic, Hindi, Turkish, and Japanese.
- Free for research under CC-BY-NC; compatible with Transformers, vLLM, SGLang, llama.cpp, and Ollama.
- Superseded by Aya Expanse 8B for new projects; use Aya-23-8B for reproducibility with the original 2024 technical report.
Aya-23-8B is an open-weights, instruction-tuned language model from Cohere Labs built to serve 23 languages at a quality level that most multilingual models have never reached. It sits at 8 billion parameters, runs on consumer-grade hardware, and consistently outperforms same-size models like Mistral-7B and Gemma-1.1-7B across translation, summarization, reasoning, and open-ended generation tasks in languages far beyond English.
The problem it actually solves
Most LLM progress has been English-first. Models like Mistral and Gemma are heavily used by multilingual communities, but they were never explicitly trained for it. The result is a steep performance cliff the moment you leave high-resource European languages. Multilingual efforts like Aya 101, BLOOMZ, and mT0 made strides in expanding access to NLP for the world, but significant room for improvement remained relative to first-class citizen languages like English and Chinese.
The previous Aya model (Aya 101) tried to solve this with sheer breadth, covering 101 languages. But breadth came at a cost. Aya 23 is an experiment in depth vs breadth, exploring the impact of allocating more capacity to fewer languages that are included during pre-training. The bet paid off.
What's under the hood
Aya-23-8B is a multilingual language model that supports 23 languages and builds on the Command family architecture, integrating the Aya Collection dataset. The base model is Cohere's Command R, which was pre-trained on data spanning all 23 target languages, giving the model a strong multilingual foundation before any fine-tuning begins.
The architecture uses several modern design choices from the Aya 23 technical report:
- Grouped Query Attention (GQA): Multiple query heads share a single key-value head, cutting inference-time memory usage significantly.
- SwiGLU activations: A gating mechanism in the feed-forward layers that consistently improves downstream task performance over standard ReLU.
- Rotary Positional Embeddings (RoPE): Enables better generalization to longer contexts than alternatives like ALiBi.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
