Skip to main content
Back

LS-Surg Segments Surgical Tools Without a Single Pixel Label

A new framework trains surgical tool segmenters using only presence labels that can be auto-extracted from video overlays, cutting pixel annotation costs.

Pro

Paper, Full Access
  • LS-Surg segments surgical tools using only per-frame presence labels that OCR can extract from robotic video overlays.
  • Built on the Multi-Class Token Transformer, extended with three temporal constraints and SAM-refined pseudo masks.
  • Temporal equivariance warps features between adjacent frames to enforce pixel-level consistency across time.
  • Class-aware semantic continuity contrasts uncertain local crops against global class tokens to activate non-salient tool parts.
  • On Cholec80 it reaches 82.08 IoU_C and 73.49 AP, within roughly 3 points of full supervision.
  • Full paper on arXiv; instance-wise annotations available on request from the authors.

Training a segmentation model to recognize surgical instruments in laparoscopic video normally requires a clinician to trace every scalpel, forcep, and clipper across thousands of frames. A team from the University of Science and Technology of China and PLA General Hospital has proposed a workaround that leans on a much cheaper signal already sitting in most robotic surgery recordings: the on-screen text listing which tools are currently in use.

Their method, LS-Surg, tackles what the authors call label-supervised surgical instrument segmentation, where the only supervision available is a per-frame yes/no list of which instruments are present. Robotic surgical videos often record both video streams and instrument presence annotations, offering substantial potential to cut manual annotation costs. Those labels can be pulled automatically from the recorded tags via OCR, no clinician required.

Why presence labels alone fall short

Knowing that a bipolar forceps appears somewhere in a frame tells you nothing about where its pixels sit. Label-supervised surgical instrument segmentation remains an under-constrained and challenging task. Prior weakly supervised work in the surgical domain has mostly relied on richer cues such as scribbles or bounding boxes, or has been limited to detection and localization rather than dense segmentation.