---
title: "Duke Bakes Grammar Into DA-Cramming to Beat BERT on One GPU"
subtitle: "A dual-stage framework injects syntactic dependency information directly into single-GPU BERT pretraining, adding 0.83 GLUE points at no extra compute cost."
url: "https://alphasignal.ai/news/duke-bakes-grammar-into-da-cramming-to-beat-bert-on-one-gpu"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-04-15T00:00:00.000Z"
updated: "2026-08-23T21:04:18.275Z"
type: "Paper"
topics:
  - "Training Infra"
  - "Llms"
subtopics:
  - "Pretraining"
  - "Scaling Laws"
  - "Small Models"
upvotes: 2599
source_url: "https://arxiv.org/abs/2311.04799"
comments: 0
truncated: true
---

# Duke Bakes Grammar Into DA-Cramming to Beat BERT on One GPU

> A dual-stage framework injects syntactic dependency information directly into single-GPU BERT pretraining, adding 0.83 GLUE points at no extra compute cost.

A Duke University team has published [DA-Cramming](https://arxiv.org/abs/2311.04799), a method that squeezes better language understanding out of a BERT-style model trained on a single GPU in 24 hours. The trick is baking grammatical dependency information straight into pretraining rather than bolting it on later during fine-tuning.

The work builds on [Cramming](https://arxiv.org/abs/2212.14034), Geiping and Goldstein's 2022 recipe that showed BERT-scale pretraining could be done on one consumer GPU in a day by stripping biases from linear layers, using scaled sinusoidal positional embeddings, and tuning the architecture for throughput. That approach got close to full BERT quality but not all the way. DA-Cramming closes some of the remaining gap by teaching the model about subject-verb and verb-object relationships from step one.

## Why chunks beat individual words

Pretraining these models typically demands substantial compute, putting them out of reach for many researchers. Prior work has tried adding syntactic knowledge to language models, almost always at the fine-tuning stage. SyntaxBERT and KERMIT, for example, inject parse trees into an already-pretrained BERT. The authors argue this misses the point: foundational language understanding gets set during pretraining, so that is where structure should be introduced.

Their theoretical basis is the Lexical Approach from linguistics, which holds that fluent language is built from chunks like subject-verb pairs rather than individual words. To operationalize this, they use spaCy's dependency parser to tag every word in the training corpus with one of four agreement categories:

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [DA-Cramming: Enhancing Cost-Effective Language Model Pretraining with Dependency Agreement Integration](https://arxiv.org/abs/2311.04799)
- [Cramming: Training a Language Model on a Single GPU in One Day](https://arxiv.org/abs/2212.14034)
- [DA-Cramming full HTML paper](https://arxiv.org/html/2311.04799)