WeatherReasonSeg Exposes How Vision AI Collapses in Rain, Fog, and Snow
A new benchmark stress-tests vision-language segmentation models against rain, fog, snow, and night, exposing systematic failures tied to weather type and severity.
ProTakeaways
- WeatherReasonSeg benchmarks vision-language segmentation models under rain, fog, snow, and night conditions.
- Combines a synthetic track with controllable severity levels and a curated real-world adverse-weather set.
- Uses Qwen LLM with mask-guided prompting to build 44,721 semantically consistent query and mask pairs.
- Evaluates across five reasoning dimensions: functionality, application, structure, interactions, and requirement matching.
- Finding one: VLM performance drops monotonically as weather severity increases, with no graceful plateau.
- Finding two: different weather types cause distinct vulnerability patterns, so robustness is not a single scalar.
Reasoning segmentation, where a vision-language model picks the right pixels based on an implicit natural language query rather than a category label, has been climbing benchmarks built mostly from clean, well-lit photos. A new benchmark called WeatherReasonSeg asks a blunt follow-up: what happens when the camera is looking through rain, fog, or snow? Performance drops in ways that depend on both the weather type and its severity, and now those drops are measurable.
The clean-image blind spot
Reasoning segmentation differs from standard segmentation because the model has to output a mask conditioned on a complex, implicit query, which may demand multi-step reasoning or external world knowledge. Most current systems handle this by adding a special [SEG] token to a multimodal LLM's vocabulary, then feeding the last-layer hidden state for that token, along with the image, into a segmentor that produces the final mask.
Evaluation is where things get shaky. The dominant benchmark, ReasonSeg, is a set of image, instruction, and mask triplets designed to test reasoning-based segmentation with implicit natural language queries, and top models on the leaderboard already sit north of 65 percent gIoU. CoReS reaches 68.1% gIoU on the validation set and 65.5% on the test set, with VisionReasoner close behind. Those numbers suggest the task is nearly solved, but they were measured on postcard-quality images.