Skip to main content
Back

Colorado State Proves Vision Encoders Know Object Orientation but LLMs Can't Use It

New probing study finds vision encoders like CLIP and SigLIP do capture rotation angles within 3 degrees, shifting blame to the language model.

Pro

  • Colorado State researchers challenge the belief that CLIP-style encoders are why MLLMs fail at orientation tasks.
  • Linear ridge regressors recover object rotations from SigLIP, ViT and CLIP embeddings with MAE under 3 degrees.
  • Tested on LLaVA-OneVision, Qwen2.5-VL-7B-Instruct, LLaVA 1.5 and LLaVA 1.6 across nine image sets.
  • Prediction errors are approximately Gaussian by Kolmogorov-Smirnov tests, meaning the signal is well behaved.
  • Feature substitution shows orientation is spread diffusely across tens of thousands of embedding dimensions.
  • Code, images, and probe scripts released at github.com/anjugopinath/MLLM_Orientation.

For years, vision-language models have flunked a task that looks trivially easy for humans: telling you which way an object is pointing. Ask GPT-style multimodal systems whether a dog is standing upright or lying sideways, or by how many degrees a photo has been rotated, and they often refuse or guess 90 degrees. The prevailing explanation blamed the vision encoder, usually some flavor of CLIP, for throwing orientation away because it was trained to match images to captions rather than reason about geometry. A new paper from Colorado State University, Why MLLMs Struggle to Determine Object Orientations, tests that assumption directly and finds it wrong.

The accused encoder has an alibi

The scale of the problem is well documented. Nichols et al. tested 15 MLLMs on questions requiring granular knowledge of object orientations and found that the best systems answered only 34% of the questions correctly. On the Visual-Spatial Intelligence Benchmark, most models perform near or below chance. Prior work hypothesized that MLLMs perform well on categorical clustering of directions into left and right but fail at angular estimation because they are pretrained on CLIP-like models focused on image-text semantic alignment rather than geometric understanding.

To check whether encoders really are the bottleneck, the authors probe four open-source systems that between them use three different visual backbones: LLaVA-OneVision uses the SigLIP vision encoder, LLaVA 1.5 and 1.6 use the CLIP vision encoder, and Qwen2.5-VL-7B-Instruct uses a native Vision Transformer (ViT) backbone. The logic is simple: if orientation information is not encoded in the embedding, no downstream model, no matter how clever, can recover it.