Skip to main content
Back

Tsinghua's RGB-IR Detector Cuts FLOPs 70% While Beating Every Benchmark

A Tsinghua team recasts CLIP text embeddings as a shared alignment bridge between RGB and thermal streams, cutting fusion FLOPs 70% while topping four benchmarks.

Pro

  • Tsinghua team uses CLIP text embeddings as a shared bridge between RGB and IR, not just an auxiliary prompt (paper).
  • New bi-support modeling splits RGB-IR interaction into consensus and discrepancy channels, both dynamically recalibrated.
  • Achieves 68.9 mAP on LLVIP, 50.4 on FLIR, 81.4 mAP@0.5 on DroneVehicle, all state of the art.
  • Routing fusion through category text cuts fusion FLOPs 70.1%, from 13.56G to 4.05G, with 15.5M total params.
  • Custom dual-branch frequency-aware IR backbone runs at 302 FPS with just 1.05M parameters.
  • Code and configs are available on GitHub under a YOLO-based training pipeline.

Multispectral detectors that combine visible RGB with thermal infrared have long promised robust perception in the dark, through smoke, or against camouflage. The catch is that the two sensors do not see the world in the same way, and simply concatenating their features tends to blur the very asymmetries that make them complementary. A new framework from a team at Tsinghua and China University of Mining Technology proposes a different plumbing job: use text as a shared semantic bridge, and explicitly model the places where the two modalities agree and disagree.

The paper, Bridging the RGB-IR Gap, argues that most prior text-guided fusion methods treat language as a bolt-on prompt. The authors instead route the entire RGB-IR interaction through a small set of CLIP category embeddings, and add a mechanism they call bi-support modeling. The resulting detector sets new state of the art on FLIR, LLVIP, DroneVehicle and M3FD while running with a fraction of the parameters of comparable transformer-based fusers. Code is on GitHub.

The granularity problem nobody wants to name

Comparison of vanilla RGB-IR fusion, conditional prompt fusion, and the proposed semantic bridge fusion

RGB is dense with texture, edges, and color, while thermal imagery gives you regional heat blobs and object presence but weak local detail. FLIR is a widely used RGB-thermal benchmark for autonomous driving. LLVIP is a paired visible-infrared dataset designed for low-light vision and pedestrian perception. DroneVehicle is a UAV-based RGB-infrared benchmark for vehicle detection under aerial viewpoints. M3FD is a multiscenario benchmark that provides aligned visible-infrared image pairs. Across all of them, the same asymmetry shows up: the two streams carry information at different granularities, and standard attention-based fusion tends to average them into a mushy consensus.

The authors identify two specific failure modes: