Duke Bakes Grammar Into DA-Cramming to Beat BERT on One GPU
A dual-stage framework injects syntactic dependency information directly into single-GPU BERT pretraining, adding 0.83 GLUE points at no extra compute cost.
PRO- DA-Cramming adds syntactic dependency information to single-GPU BERT pretraining without extra compute cost.
- Four submodels encode subject-verb, verb-object, prepositional-object, and complement relationships in parallel.
- Beats Crammed BERT on 7 of 8 GLUE tasks with a 0.83-point average improvement.
- Biggest wins on RTE (+3.13%) and MRPC (+2.26%); SV submodel is the most important overall.
- Fine-tuning-stage syntax injection methods like SyntaxBERT actually hurt performance under a Cramming budget.
- Random agreement embeddings collapse GLUE to 53.1, confirming submodels encode real structural signal.
A Duke University team has published DA-Cramming, a method that squeezes better language understanding out of a BERT-style model trained on a single GPU in 24 hours. The trick is baking grammatical dependency information straight into pretraining rather than bolting it on later during fine-tuning.
The work builds on Cramming, Geiping and Goldstein's 2022 recipe that showed BERT-scale pretraining could be done on one consumer GPU in a day by stripping biases from linear layers, using scaled sinusoidal positional embeddings, and tuning the architecture for throughput. That approach got close to full BERT quality but not all the way. DA-Cramming closes some of the remaining gap by teaching the model about subject-verb and verb-object relationships from step one.
Why chunks beat individual words
Pretraining these models typically demands substantial compute, putting them out of reach for many researchers. Prior work has tried adding syntactic knowledge to language models, almost always at the fine-tuning stage. SyntaxBERT and KERMIT, for example, inject parse trees into an already-pretrained BERT. The authors argue this misses the point: foundational language understanding gets set during pretraining, so that is where structure should be introduced.
Their theoretical basis is the Lexical Approach from linguistics, which holds that fluent language is built from chunks like subject-verb pairs rather than individual words. To operationalize this, they use spaCy's dependency parser to tag every word in the training corpus with one of four agreement categories:
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
