---
title: "HKU's Multi-SPIN Boosts Edge AI Inference Goodput by 88% on Weak Devices"
subtitle: "Multi-SPIN turns every user's phone into a draft engine, boosting edge LLM token throughput by up to 88% through smart bandwidth and draft-length co-optimization."
url: "https://alphasignal.ai/news/hku-s-multi-spin-boosts-edge-ai-inference-goodput-by-88-on-weak-devices"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-06-03T00:00:00.000Z"
updated: "2026-06-11T02:41:04.468Z"
type: "Paper"
topics:
  - "Infra"
  - "Gpus"
subtopics:
  - "Inference Optimization"
  - "Edge Deployment"
upvotes: 1706
source_url: "https://arxiv.org/abs/2606.04581"
comments: 0
truncated: true
---

# HKU's Multi-SPIN Boosts Edge AI Inference Goodput by 88% on Weak Devices

> Multi-SPIN turns every user's phone into a draft engine, boosting edge LLM token throughput by up to 88% through smart bandwidth and draft-length co-optimization.

Running large language models at the edge is one of the harder unsolved problems in applied AI. The devices users carry are too weak to run a full LLM, but shipping every prompt to a remote server adds latency, cost, and privacy risk. A new paper from the University of Hong Kong proposes a clever middle ground: let the device do the cheap part, and let a nearby edge server do the expensive part, in a tightly coordinated loop.

The architecture is called [Multi-SPIN](https://arxiv.org/abs/2606.04581) (Multi-Access Speculative Inference), and it extends the well-known speculative decoding technique into a multi-user wireless setting. The result is a system that can serve multiple users simultaneously with full LLM quality, while distributing the compute burden intelligently across heterogeneous hardware.

## Why edge LLM inference is so hard

The growing gap between the increasing complexity of large language models and the limited computational budgets of edge devices poses a key challenge for efficient on-device inference, despite gradual improvements in hardware capabilities. The standard playbook has been to either compress the model aggressively or offload everything to the cloud. Existing strategies, such as aggressive quantization, pruning, or remote inference, trade accuracy for efficiency or lead to substantial cost burdens.

There is a third option that has gained traction: speculative decoding. Speculative decoding accelerates LLM inference by allowing a lightweight draft model to predict multiple future tokens that are subsequently verified by a larger target model. The key insight is that the verification step can be done in a single parallel forward pass, so if the small model guesses right most of the time, you get multiple tokens for the price of one. Speculative decoding has demonstrated remarkable improvements in the generative pipeline on high-end GPUs, achieving speedups up to 6.5x compared to traditional autoregressive decoding.

But almost all prior work assumes the draft model and target model live on the same machine. Multi-SPIN asks: what if the draft model runs on the user's phone, and the target model runs on a nearby edge server?

## The architecture: phones draft, server verifies

The Multi-SPIN architecture utilizes on-device small language models to generate and upload candidate token drafts, while an edge server operates the LLM to verify them in parallel batches. This is a natural split: small language models (SLMs) are cheap enough to run on a phone, and the edge server's GPU can verify drafts from many users at once by batching them together.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Multi-SPIN: Multi-Access Speculative Inference for Cooperative Token Generation at the Edge](https://arxiv.org/abs/2606.04581)
- [SLED: A Speculative LLM Decoding Framework for Efficient Edge Serving](https://arxiv.org/html/2506.09397v3)
- [Efficient LLM Inference over Heterogeneous Edge Networks with Speculative Decoding](https://arxiv.org/abs/2510.11331)
- [GoodSpeed: Optimizing Fair Goodput with Adaptive Speculative Decoding in Distributed Edge Inference](https://arxiv.org/abs/2512.09963)
- [Optimizing Speculative Decoding for Serving Large Language Models Using Goodput](https://arxiv.org/html/2406.14066v2/)