Skip to main content
Back

Shanghai Jiao Tong's VL-RouterBench Cuts Multimodal AI Costs by 60% Without Losing Accuracy

A new benchmark tests 10 routing methods across 17 vision-language models, revealing routers can slash costs by 5x while nearly matching the strongest single model.

Pro

  • VL-RouterBench introduces the first reproducible benchmark for vision-language model routing across 14 datasets and 17 models.
  • Covers 30,540 samples and 519,180 sample-model pairs with real token-based cost matrices.
  • RouterDC matches strongest single model accuracy at roughly one-fifth the inference cost.
  • Oracle reaches 95.6% accuracy at $0.37 per 10K samples, showing large headroom for router improvement.
  • BGE-M3 plus SigLIP-L-16 with normalized concatenation is the best feature-level configuration.
  • Full pipeline is open-sourced on top of VLMEvalKit for reproducible routing research.

Picking which vision-language model to call on any given request has quietly become one of the trickiest engineering problems in multimodal deployment. A 1B model handles a simple OCR crop just fine, but a chart-heavy STEM question probably needs a 72B behemoth. A team from Shanghai Jiao Tong University and HKUST is trying to make this decision measurable with VL-RouterBench, the first large-scale, reproducible benchmark aimed specifically at routing between vision-language models (VLMs).

Why VLM routing needs its own scoreboard

Routing research for large language models is maturing, but the vision-language domain has lacked a systematic and reproducible benchmark of its own. VLM routing also faces pressures that text routing does not: task types vary wildly across visual question answering, chart OCR, and visual reasoning, and different VLMs vary substantially in modality interaction and semantic representation. Both factors compound the difficulty of designing and evaluating a router.

The stakes are practical. Commercial systems like GPT-5 already ship with built-in routers that pick between underlying models, yet there has been no shared yardstick for measuring whether such a router is any good at multimodal decisions.

Inside the benchmark