Tencent's ncnn Brings FlashAttention and LLMs to Mobile Hardware
Tencent ships ncnn 20260526 with Vulkan FlashAttention, broad BF16 coverage across all CPU backends, and first-ever HarmonyOS prebuilts
Takeaways
- ncnn 20260526 released: Tencent ships a major update to its open-source on-device inference framework with strong LLM-focused additions.
- Vulkan FlashAttention: Native GPU SDPA/FlashAttention implementation lands, plus RotaryEmbed, GroupNorm, and other Transformer operators previously missing from the GPU path.
- BF16 everywhere: Comprehensive BF16 storage paths added across x86, ARM (ARMv8.4), RISC-V, MIPS, and LoongArch backends.
- Large model loading: Persistent Vulkan pipeline cache, memory-mapped model loading, and host-memory weight residency cut startup time and peak RAM for large models.
- LLM benchmark tool: New
benchncnn_llmships with 7 built-in small LLM configs (Qwen3, Llama 3.2, MiniCPM4, etc.) and outputs tokens-per-second. - HarmonyOS prebuilts: First official HarmonyOS precompiled packages (CPU/Vulkan, static/dynamic) added to the release; download here.
Tencent has shipped ncnn release 20260526, the latest versioned drop of its open-source on-device inference framework. The release lands with a notably different character than previous ones: while ncnn has always been a mobile-first CNN runner, this update pushes hard into large language model territory, adding Transformer-specific operators, LLM benchmarking tooling, and a suite of memory optimizations that make running multi-billion-parameter models on consumer hardware far more practical.
The headline: Vulkan gets FlashAttention
The biggest single addition is a native Vulkan implementation of SDPA (Scaled Dot-Product Attention) with a FlashAttention path. FlashAttention is an algorithm that rewrites the attention computation so that intermediate results stay in fast on-chip memory rather than being written out to slower GPU memory -- this dramatically reduces memory bandwidth usage during inference, which is the primary bottleneck when running Transformer models on mobile GPUs.
ncnn is a high-performance neural network inference framework optimized for mobile and other platforms developed by Tencent. The new Vulkan SDPA layer ships with 2x2 unroll and local memory optimizations, plus two FlashAttention implementations (cooperative matrix and non-cooperative matrix paths) that support attention masks and KV cache concatenation. Alongside SDPA, the GPU backend gains RotaryEmbed, GroupNorm, Reduction, Unfold, Softplus, and Shrink operators -- all previously falling back to CPU, which broke the execution graph for Transformer models running on Vulkan.
BF16 lands everywhere
BF16 (brain float 16) is a 16-bit number format that keeps the same dynamic range as 32-bit floats but halves memory usage. It has become the dominant training and inference format for large models. This release makes BF16 a first-class citizen across all of ncnn's CPU backends.