Meta's JiTTests Catches 74 Privacy Bugs Before They Hit Production
Meta's LLM-powered test generator writes failing tests that prove bugs exist, catching 74 privacy incidents and cutting privacy SEVs 20 percent.
- Meta deployed Catching JiTTests to production, using LLMs to generate failing tests that prove a diff introduces a behavior change.
- In H1 2026 it caught 74 privacy SEVs at the diff stage and cut overall privacy SEV occurrence by 20%.
- 589 engineers (64% of outreach) engaged with findings, reporting a 60% true-positive rate.
- Two parallel workflows: dodgy diff (treat diff as a mutant) and intent-aware (infer intent, then construct risk mutants).
- False positives filtered via LLM-as-Judge ensemble plus deterministic rule-based RubFake, then ranked for escalation.
- Full details in the FSE 2026 paper; system runs overnight on spare capacity on risky diffs.
Meta has moved Catching Just-in-Time Tests, or JiTTests, from research prototype to its production code-review pipeline, according to a company report. The system examines risky diffs, generates executable unit tests, and runs them before merge. A successful catch produces a test that passes on the parent version and fails on the proposed change, giving the author a reproducible behavioral difference to investigate.
| Measure | Result |
|---|---|
| Engineer engagement | 589 engineers, representing 64% of those contacted |
| Reported true-positive rate | 60% of surfaced findings |
| Serious privacy bugs caught | 74 before reaching production |
| Privacy incident rate | 20% lower occurrence of privacy SEVs |
A privacy SEV is an incident tracked through Meta’s severity process. The figures come from internal production telemetry and engineer responses. Meta associates JiTTests with the reduction in privacy SEVs, although the published results do not isolate how much of that decline came from the system alone.
Tests that make a diff prove itself
Conventional LLM code-review tools inspect a patch and produce comments about possible faults. JiTTests generates a unit test that demonstrates an observable change. In its terminology, the parent is the code before the diff and the child is the proposed version.
When a generated test passes on the parent and fails on the child, the author can reproduce the result and decide whether the changed behavior was intended. Existing test suites remain useful for known requirements, while JiTTests targets risks introduced by a specific edit.
Two routes to a failing test
Mutation testing deliberately injects small faults, called mutants, and checks whether tests detect them. JiTTests adapts that technique through two generation strategies described in Meta’s FSE 2026 paper:
- Dodgy diff workflow: The system treats the submitted diff as a mutant. An LLM generates tests that distinguish its behavior from the parent version, searching broadly without inferring the author’s intent.
- Intent-aware workflow: An LLM infers the intended change from the code, title, and summary. It identifies implementation risks, creates mutants representing those possible faults, and generates tests designed to expose them. The resulting tests are then evaluated against the real diff.
Both workflows convert an LLM hypothesis into code that can be compiled and executed. That execution step supplies the failure message, trace, and parent-child comparison used by the rest of the pipeline.
A gauntlet for weak catches
Every generated test that separates the parent and child begins as a weak catch because the difference may be intended or irrelevant. Meta filters these candidates before contacting an engineer:
- LLM-as-Judge ensemble: Multiple models assess the test code, execution result, failure message, stack trace, diff, and inferred intent. Their votes estimate whether the catch represents a real defect.
- RubFake: A rule-based filter rejects known false-positive patterns with deterministic checks. Its stable judgments complement the semantic assessment from the model ensemble.
The combined signals rank each catch, and only the highest-ranked findings trigger outreach. JiTTests runs overnight on spare compute capacity, processing diffs submitted the previous day that a risk-scoring system has selected for analysis.
Aggregate supervision keeps outreach bounded
Meta describes the operating model as “human on the loop.” Engineers define criteria, monitor aggregate performance, and adjust the system based on outcomes. Individual authors receive only the findings that survive generation, execution, filtering, and ranking.
When contacted, an author inspects the observed failure and decides whether the behavior change was intended. Meta says this workflow often lets engineers make that decision from the result without reading the generated test code in detail. The engagement and true-positive rates provide feedback for tuning thresholds and controlling notification volume.
Costs, coverage, and open questions
Privacy and security regressions fit this approach because a small edit can alter behavior with consequences that are difficult to infer from syntax alone. Parent-child execution can expose those changes even when the generated explanation is imperfect.
- Compute cost: Test generation, compilation, and execution require more resources than producing a review comment.
- Triage noise: A 60% true-positive rate leaves four false or unhelpful findings for every ten surfaced.
- Risk-score dependence: The pipeline examines selected diffs, so its coverage depends on the upstream system identifying risky changes.
- Execution constraints: Adopters need build environments that can run generated tests safely and reproduce both parent and child revisions.
- Portability: Meta has not released JiTTests as open source. External teams must recreate the pipeline and connect it to their own review, build, and test infrastructure.
Teams evaluating the design will also need local measurements for compute cost per diff, supported languages, test latency, false-negative rates, and the notification threshold developers will tolerate. Those values determine whether the approach works beyond a narrow set of high-risk changes.
A practical build order
- Select a constrained bug class with measurable production outcomes.
- Prioritize diffs using repository, ownership, and change-risk signals.
- Generate tests against reproducible parent and child revisions.
- Capture test code, exit status, failure output, traces, and inferred intent.
- Combine semantic model judgments with deterministic false-positive rules.
- Rank findings and enforce a limited outreach budget.
- Track engagement, confirmed defects, rejected findings, and escaped incidents.
JiTTests offers a concrete pattern for agentic development tools: produce executable artifacts, verify them against real code, filter aggressively, and reserve human attention for the strongest results.
Comments
