MIT Trains AI on 3.8M Student Code Traces to Understand How Coders Think
A 3.8M-trace dataset from Pencil Code shows that training on real student edit histories teaches models to predict who wrote code, not just what.
Takeaways
- New 3.8M-trace dataset from Pencil Code captures full student coding sessions across 9 years and 1.3M users.
- Models trained on real traces beat synthetic-trace and final-program baselines on diversity and behavior prediction.
- A per-student embedding layer encodes individual style: comment counts, backtracking frequency, color choices.
- Trace model recovers broken student code to a working state over 60% of the time versus 39% for synthetic.
- New students personalize efficiently: just 2 to 4 finetuning traces per learner before scores plateau.
- Code and gated dataset released on GitHub and Hugging Face.
Most code datasets show you the destination: the final, working program someone submitted. What they hide is the journey, all those broken attempts, weird detours, and stylistic quirks that reveal how a person actually thinks. A new paper from MIT CSAIL, Stanford, and the University of Florida argues that this missing signal is exactly what language models need if they want to understand coders, not just code.
The team released a dataset of 3.8 million programming traces from students learning on Pencil Code, a free educational platform, and trained models on full edit histories rather than finished submissions. The resulting models can predict when a specific student will backtrack, how many comments they tend to write, and how to nudge a broken program back to working while preserving the learner's personal style.
Why finished code is the wrong training signal
Current training data captures what users produce, but not how they produce it, and as a result, state-of-the-art models often adopt non-human-like problem-solving strategies. That is fine if your only goal is to spit out correct code, but it becomes a serious limitation for educational tools, where the whole point is understanding a learner's process, catching them mid-mistake, and giving hints that fit their skill level.
Novice programmers in particular produce traces that reflect diverse reasoning: exploratory behavior to understand a concept, re-strategizing in response to bugs, and personalizing stylistic choices. Prior public datasets either recorded extremely granular IDE keystrokes or only kept final submissions, leaving a gap for anything you could actually pretrain a language model on.
Inside the 3.8M trace dump
Pencil Code is an open-source educational platform supporting creative projects spanning turtle graphics, music composition, speech synthesis, and interactive storytelling. Its dual-modality editor lets users write code through either a visual block-based interface (similar to Scratch) or directly in languages like CoffeeScript and JavaScript. The authors constructed a dataset of 3.8M programs written on Pencil Code from 2015 to 2024, with each trace consisting of a hashed student ID, an assignment title such as snowman, and an ordered sequence of programs written by the student along with timestamps.