Skip to main content
Back

Sony and Alberta's RESP Catches Video Game Glitches Without Any Fine-Tuning

A new framework from University of Alberta and Sony Interactive Entertainment uses reference frames to help VLMs catch subtle video game glitches they'd otherwise miss.

Pro

  • RESP pairs each test frame with an earlier reference frame from the same video for VLM-based glitch detection, no fine-tuning required. Paper
  • Qwen3-VL-8B gains +0.32 F1 and +0.49 recall on RefGlitch when given a reference; GPT-5 gains +0.25 F1.
  • LastCleanFrame (most recent predicted-clean frame) is the best automatic reference selection policy and generalizes across five VLMs.
  • New RefGlitch dataset: 1,000 labeled reference/test pairs in Godot across missing object, clipping, floating, corrupted texture, and lighting glitches.
  • Video-level logistic regression aggregator over engineered features adds +0.10 accuracy and +0.07 F1 on PhysGame plus VideoGameQA-Bench.
  • Code and data released at github.com/PipiZong/RESP_code.

Spotting a missing arm on a game character, a texture that has gone haywire, or a character floating a few inches off the ground sounds trivial for a modern vision-language model. It isn't. Show a VLM a single screenshot and it will often shrug and call the scene glitch-free, because there is no reference for what the shot should look like. A team from the University of Alberta and Sony Interactive Entertainment has a fix, and it requires no fine-tuning.

Their framework, RESP (Reference-guided Sequential Prompting), pairs each test frame with an earlier one from the same video and lets the VLM decide whether anything unnatural has changed. The core move is reframing detection as within-video comparison rather than isolated classification, then aggregating noisy per-frame predictions into a stable clip-level decision without touching the model weights.

Why single-frame QA keeps failing

Modern games ship frequent builds across platforms, hardware, and graphics settings, and manual QA cannot keep pace. Prior automated approaches mostly treat glitch detection as image classification: feed one screenshot, ask if it is broken. That paradigm has a fundamental blind spot. A missing robot arm, a subtle lighting shift, or a character hovering an inch above the floor is only obvious if you know what a normal frame looks like.

The authors show this concretely. GPT-5 predicts a missing-object test frame as glitch-free when evaluated in isolation, because the absent robot arm is not obvious without a reference. Give it a reference frame with the arm intact and it correctly flags the anomaly. Existing multi-frame efforts either rely on streamer transcripts as auxiliary metadata (unavailable during development) or require expensive instruction tuning that still underperforms in the wild.

How RESP works