Skip to main content
Back

XYZ AI Lab's Aquila-mini Runs Locally on a Single 24GB GPU

A 35B open-weight Deep Search agent that tops the sub-40B leaderboard now ships in GGUF quants runnable on a single consumer GPU.

  • GGUF quants of XYZ-Aquila-mini now available for local llama.cpp, Ollama, and LM Studio use
  • 35B MoE (~3B active) post-trained from Qwen3.6-35B-A3B for agentic Deep Search
  • Tops every sub-40B benchmark: 97.1 GAIA, 78.8 BrowseComp, 51.1 HLE
  • Runs on a single 24GB GPU at Q4_K_M (21.4GB); IQ2 quants fit in 10-12GB
  • Requires the AxisAgentic harness for search, scrape, and Python tools
  • Apache-2.0 license, 256K context, bilingual English/Chinese web browsing

A community contributor has published a GGUF conversion of XYZ AI Lab’s Aquila-mini Deep Search agent. The release supports llama.cpp and GGUF-compatible applications such as LM Studio, Ollama, and Jan, allowing reduced-precision inference on a workstation instead of the upstream eight-GPU SGLang configuration.

The upstream Aquila-mini checkpoint remains a 35-billion-parameter mixture-of-experts model. Quantization reduces its storage and memory requirements, with files ranging from 69.4GB in BF16 to 9.78GB in IQ2_XXS. A 21.4GB Q4_K_M file can fit its weights on many 24GB GPUs, subject to runtime overhead and the memory reserved for context.

A small active path through a 35B model

Aquila-mini activates roughly 3 billion parameters for each token while retaining about 35 billion parameters across all experts. That sparse architecture reduces computation per token, while the complete set of expert weights still has to reside in GPU memory, system RAM, or both. This distinction explains why the model can have relatively modest compute requirements yet still produce a 21.4GB Q4 quant.

XYZ AI Lab post-trained the model from Qwen3.6-35B-A3B using its bounded-exploration AI4AI pipeline. Humans specify capabilities, evidence, constraints, risk limits, and acceptance criteria; automated agents then diagnose failures and propose scoped changes to training data, post-training, tools, runtime behavior, context management, evaluation, and infrastructure.

The resulting checkpoint targets long-horizon research tasks rather than unrestricted text generation. Its documented capabilities include English and Chinese web browsing, multi-source evidence collection, source verification, planning, and recovery after failed tool interactions.