FlashAttention-V Hits 42x Speedup Bringing AI to RISC-V Edge Chips
A redesigned FlashAttention kernel unlocks long vector registers on RISC-V and Arm SVE, delivering up to 42x speedup on CPU inference.
PRO- FlashAttention-V redesigns the attention kernel to exploit vector lengths beyond the head dimension on RISC-V and Arm SVE.
- Inter-head packing maps multiple attention heads into a single vector register using loop reordering and unrolling.
- Simulated speedup reaches 22x to 42x over scalar FlashAttention in prefill at 512-bit VL, plus 2x to 2.5x at 64 lanes.
- Decode gets 8x to 11x, but wider vectors help less due to single-token memory-bound execution.
- Implementation lives in llama.cpp, tested on TinyLlama, Llama 3.2, Qwen2.5, and Pythia-410M.
- Q8_0 quantization layout blocks long-vector scaling: packing and reduction consume 60% of cycles at 2048-bit VL.
Running transformer inference on CPUs used to feel like a compromise. That is changing fast, especially for small language models under 10B parameters that get deployed on edge devices, embedded boards, and increasingly capable RISC-V and Arm chips. One stubborn bottleneck remains: the attention kernel. A new paper introduces FlashAttention-V, a redesigned version of FlashAttention built specifically for scalable vector architectures, and the reported speedups are substantial.
Why FlashAttention stalls on vector CPUs
FlashAttention is an optimized implementation of self-attention that reduces memory usage and improves computational efficiency by tiling the computation across blocks, maintaining running maxima and normalization factors for numerically stable on-the-fly softmax. It was designed for GPUs, though the algorithm has since been ported to CPU stacks like llama.cpp and its underlying ggml tensor library.
The trouble appears on modern vector-length-agnostic ISAs like the RISC-V Vector Extension (RVV) and Arm SVE, where a single binary can run on hardware with wildly different vector register widths. Existing implementations on RISC-V, Arm, and x86 tie the vector length (VL) to the head dimension D. The vectorized FlashAttention kernels in llama.cpp are all limited to VL less than or equal to D. Head dimensions are typically 64 or 128, which caps useful vector length at roughly 1024 or 2048 bits in FP16. Any wider register just sits idle.
That is a real limitation because upcoming vector processors like Ara2, AraXL, and Vitruvius+ target vector widths of 4096 bits and beyond. The parallelism is there in the hardware. FlashAttention cannot reach it.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
