Skip to main content
Back

TalkSketchD Shows Designers' Mumbling Beats Sketches for AI Intent Alignment

A small study shows that feeding a designer's messy real-time speech alongside their rough sketches lifts intent-alignment scores from a median of 2 to 7 out of 7.

Pro

Paper, Full Access
  • TalkSketchD pairs freehand sketches with time-aligned spontaneous speech from 11 designers doing toaster ideation.
  • 58 instances, iPad Pro plus Apple Pencil, transcripts kept raw with errors intact.
  • Sketch-only prompts to Gemini 2.5 Flash Image scored a median 2/7 on intent alignment.
  • Adding the messy live transcript jumped scores to a median 7/7 across form, function, experience.
  • Mann-Whitney U tests confirmed the gap with p under 10 to the minus 14 on all metrics.
  • Transcription errors like "tree" for "three" still leak into generated images as visual artifacts.

Designers rarely sketch in silence. They mumble, narrate, and rethink out loud while their pencil moves, and much of what they actually mean never makes it onto the paper. A new dataset called TalkSketchD tries to capture that missing channel, and an accompanying experiment shows that piping the raw, error-ridden speech transcript into a multimodal LLM substantially improves how well sketch-to-image generation reflects what the designer had in mind.

The gap between drawn and meant

Early-stage ideation often relies on rough sketches created under time pressure, leaving much of the designer's intent implicit. In practice, designers speak while sketching, verbally articulating functional goals and ideas that are hard to express on paper. Most sketch-based AI tools today ignore that verbal channel entirely, and users notice. In an earlier study using off-the-shelf tools, one participant described the results as "kind of crazy," another concluded they might have to redraw the whole thing so the AI could understand, and a third said it took too much time and they would rather just draw.

The authors argue that spontaneous speech, even when fragmented and full of transcription errors, carries the functional and experiential intent that a hurried sketch cannot.