Skip to main content
Back

ETS Montreal Proves AI Still Can't Read Human Ambivalence in Health Apps

A new study probes whether deep learning can spot the subtle mixed feelings that make people abandon health interventions, and finds today's models are not ready.

Pro

  • New study benchmarks deep learning for detecting ambivalence and hesitancy in health intervention videos, using the BAH dataset.
  • Three setups tested: supervised learning, unsupervised domain adaptation for personalization, and zero-shot inference with multimodal LLMs.
  • Best supervised video model (Video-FocalNet base) reached 0.5663 AVGF1 and 0.6732 AP, well below usable.
  • Fine-tuned AffectGPT hit 0.7229 AVGF1 at video level, showing MLLMs plus transcripts are the strongest baseline.
  • Text modality dominates; naive fusion of all three streams sometimes hurts, because A/H is a cross-modal conflict.
  • Code released at github.com/sbelharbi/ah-digital-health-interventions with recommendations for a two level affect-then-conflict architecture.

Behavior change is hard, and one of the biggest reasons people quit a diet, a medication plan, or an exercise app is not laziness. It is ambivalence: the fuzzy, in-between state where you kind of want to change and kind of do not. A team from ETS Montreal and Concordia University has published a study asking whether modern deep learning can actually detect this state from video, and the honest answer is: not yet. Their paper benchmarks supervised learning, domain adaptation, and multimodal LLMs on the task and lays out why it is unusually hard.

Why mixed feelings matter for digital health

Ambivalence and hesitancy (A/H) are the primary reasons individuals delay, avoid, or abandon health behavior changes, and they surface as affective inconsistency across modalities like facial expressions, tone, verbal cues, and body language. In an in-person session, a trained clinician picks up on these signals and adjusts. In an app, nobody is watching.

That gap matters because face-to-face care is costly, has limited accessibility, and is difficult to scale, while digital interventions promise a cheaper route through web and phone-based apps. If a model can flag ambivalence in real time, the app can change strategies, offer support, or reframe the message. That is the pitch.

Digital behavior change intervention pipeline with A/H detection

The BAH dataset, and why the task is brutal

The experiments use the BAH dataset, composed of videos from 300 participants recorded in their own environments while answering seven questions, some designed to elicit A/H, with visual, audio, and transcript modalities annotated at frame and video level, totaling 1,427 videos and 10.60 hours of recording. Frames are cropped and aligned with RetinaFace, audio is turned into log Mel spectrograms via VGGish, and transcripts come from Whisper large v3 with BERT word embeddings.