Skip to main content
Back

TUM Lets ChatGPT-4o Run Eye-Tracking Pipelines, Beating Hand-Tuned Baselines

A new framework turns eye-tracking analysis into a natural-language conversation, letting non-programmers run classical fixation and saccade detectors through prompts to ChatGPT-4o.

Paper, Full Access
  • TUM researchers built a code-free eye-tracking pipeline where ChatGPT-4o generates I-VT and I-DT detectors from natural-language prompts.
  • The system runs three staged LLM calls: schema inference, preprocessing plus detector synthesis, and diagnostic feedback for iterative refinement.
  • On GazeCom, the LLM-configured I-VT hit 0.973 fixation F1 and 0.847 saccade F1, beating hand-tuned baselines and 1D CNN-BLSTM.
  • Simple prompts matched detailed prompts within 0.002 F1, supporting the beginner-friendly claim across the ablation.
  • First-run failures on Hollywood2_em and pair-programming data exposed prompt sensitivity, though diagnostic feedback recovered performance on retry.
  • Full prompt templates are released in the appendix; no accompanying code repository was published.

Eye-tracking research has a plumbing problem. The classical algorithms that segment raw gaze streams into fixations and saccades are decades old and well-understood, but running them means wrangling heterogeneous file formats, unit conversions, sampling rates, and threshold parameters that break silently when any assumption shifts. A new paper from TUM's Munich Center for Machine Learning proposes to skip the code entirely and let an LLM drive the pipeline from plain-English prompts.

The framework, described in this preprint, uses ChatGPT-4o as an orchestrator that inspects raw eye-tracking files, generates cleaning and detection code on the fly, and produces diagnostic reports the user can iterate on by editing the prompt.

The onboarding tax on gaze research

Raw fixation data consist of continuous coordinate streams, which must be semantically segmented into discrete events such as fixations and saccades to become interpretable. Two algorithms dominate this segmentation: I-VT (velocity-threshold identification), which labels a sample as a saccade when angular velocity crosses a cutoff, and I-DT (dispersion-threshold identification), which treats a fixation as a temporal window where gaze stays within a small spatial region.

Both are simple in theory and brittle in practice. Classical detectors work well when tuned, yet remain highly sensitive to preprocessing and parameterization, which limits their usability outside specialized laboratories. Get the sampling rate wrong, mistake pixels for degrees of visual angle, or pick a velocity threshold that doesn't match the tracker's noise floor, and the output collapses into either all-fixations or all-saccades.