GitHub Ditches Its LLM for ModernBERT, Cutting Secret Scan Costs by 98%
GitHub swapped its LLM-based secret scanner for a fine-tuned ModernBERT classifier, boosting acceptance 11.5%, cutting scan time 86%, and serving costs 98%.
- GitHub replaced its LLM-based secret scanner with a fine-tuned ModernBERT classifier for generic secret detection.
- Repository owner-weighted alert acceptance rose 11.5% in production A/B testing against the previous pipeline.
- Median post-push scan execution time dropped 86%; serving cost dropped more than 98%.
- Offline evaluation showed 13% higher precision and 19% higher recall than the LLM pipeline.
- Speed gains enabled extending generic secret detection to push protection, blocking leaks before commit.
- Model exported to ONNX with 8-bit quantization; two-stage regex-then-classifier pipeline keeps deterministic recall.
GitHub has replaced the large language model behind its generic secret scanner with a smaller, fine-tuned ModernBERT encoder. According to GitHub’s technical post, the new classifier finds more exposed credentials, produces fewer false positives, responds faster, and costs substantially less to serve.
GitHub evaluated the new and previous pipelines against the same public post-push traffic, then ran a customer A/B experiment to measure how repository owners handled the resulting alerts.
| Metric | Reported change | Evaluation |
|---|---|---|
| Precision | 13% higher | Same public post-push traffic |
| Recall | 19% higher | Same public post-push traffic |
| Owner-weighted acceptance | 11.5% higher | Customer A/B experiment |
| Median scan execution time | 86% lower | Production pipeline |
| Serving cost | More than 98% lower | Production pipeline |
Precision measures how many flagged candidates are genuine secrets. Recall measures how many known secrets the scanner finds. GitHub weighted acceptance by repository owner so customers resolving large batches of alerts could not dominate the A/B result.
Why regex mistakes hashes for keys
Many provider-issued credentials have a recognizable prefix, fixed length, restricted alphabet, or checksum. Generic secrets lack those reliable signatures, leaving scanners to distinguish credentials from build IDs, asset hashes, test values, and tutorial placeholders that share the same surface patterns.
Security teams call these look-alikes hard negatives: values that match a credential pattern but cannot be used as credentials. Additional regex exceptions can suppress known false positives, but every exception can also hide a real leak.
A fast filter, followed by context
GitHub kept regex as the high-recall front end and assigned ambiguous candidates to ModernBERT, preserving inexpensive pattern matching while using local context to reduce false positives. The production pipeline has two stages:
- Candidate generation: Regex filters scan repository content for values that resemble secrets. This stage favors recall because any missed candidate never reaches the classifier.
- Context-aware classification: ModernBERT evaluates each candidate with its file path, proposed secret type, and nearby code or text, then predicts whether the value represents a real credential.
The encoder maps its input to a numerical representation and returns a classification, avoiding the computation required to generate natural-language output. Context such as example, dummy, fake, or replace-me can help identify documentation and placeholders, while an assignment to an environment variable in a deployment file can signal a live credential.
Training on the scanner’s mistakes
GitHub assembled training data around the distinctions that regularly confuse generic secret detectors, combining production-derived examples, broad regex matches, and generated edge cases:
- Product-derived examples: Repository samples validated by people.
- High-recall candidates: Regex matches labeled through human annotation and weak supervision, where rules or other signals infer labels at scale.
- Synthetic examples: Generated cases covering documentation, test fixtures, checksums, placeholders, and near-miss credential pairs.
Synthetic rows pass deterministic checks and model-based quality review before entering the dataset. Each model input contains the candidate value, file path, proposed secret type, and a bounded window of surrounding content.
GitHub grouped rows sharing a repository, candidate value, or source example before splitting the dataset. That grouping keeps near-duplicates out of both training and evaluation, reducing the risk of inflated results caused by data leakage.
For production serving, GitHub exports the trained model to ONNX and applies 8-bit quantization. ONNX provides a portable inference format, while quantization stores model values with fewer bits to reduce memory use and computation.
Lower latency moves detection into the push path
An 86% reduction in median execution time made generic secret detection practical during a push, allowing GitHub to return a result before accepting the changes. Previously, the heavier LLM pipeline limited this detection primarily to scans performed after code had reached a repository.
- Push protection: Evaluates changes during a push and can block exposed credentials before they enter the repository.
- Post-push scanning: Continuously examines repository content and opens alerts for developers and security teams to investigate.
Teams already using secret scanning or push protection docs should encounter the classifier through faster results and fewer false-positive alerts. GitHub’s secret protection analysis connects the work to rising code volume from AI-assisted development and the corresponding growth in accidentally committed credentials.
A narrow model fits a narrow job
GitHub’s roughly 100-million-parameter encoder outperformed its general-purpose LLM pipeline across precision, recall, latency, and serving cost. The task has characteristics that favor a specialized classifier: a measurable output, abundant labeled examples, useful local context, and high request volume.
This architecture fits other large-scale classification systems that can generate candidates cheaply and resolve ambiguity from nearby context. Its performance depends on representative hard negatives, leakage-resistant evaluation, and model packaging designed for production inference.
Comments
