Ai2's ModSleuth Automatically Maps the Hidden Supply Chain Behind Open-Source LLMs

When you download an open-source LLM, you're not just getting a model trained on text. You're getting the downstream product of dozens of other models, hundreds of datasets, and a web of upstream decisions that nobody has ever mapped end-to-end. ModSleuth, a new agentic system from the Allen Institute for AI (Ai2), is the first tool designed to automatically reconstruct that entire family tree from public artifacts alone.

The dependency problem nobody was counting

Modern LLMs don't just learn from human-written text anymore. They rely on other models to generate synthetic training data, filter and clean corpora, judge output quality, run OCR on documents, and guide reinforcement learning. Each of those upstream models has its own dependencies. The result is a recursive, multi-layered supply chain that no human team can trace manually.

The numbers make this concrete. Applying ModSleuth to four public-artifact-rich LLM releases, the researchers recovered 1,060 source-verified dependencies and constructed large-scale dependency graphs of modern LLM development. Breaking that down by model: OLMo 3 alone has 89 model dependencies and 183 dataset dependencies. Nemotron 3 Super has 273 model dependencies and 560 dataset dependencies. Some of those chains go 8 hops deep.

How ModSleuth actually works

ModSleuth is an agentic system that recursively reconstructs LLM dependency graphs from public artifacts with source-grounded evidence. That means it doesn't just read a model card and call it done. It reads papers, model cards, dataset cards, code configs, and upstream release artifacts, then follows every reference it finds, recursively, until the graph is complete.

The core technical challenge turned out to be less about reading documents and more about making sense of them. The primary challenge is no longer information extraction, but defining what constitutes a dependency and reconciling artifact references across inconsistent documentation. The same model might be referenced by five different names across five different repos. ModSleuth addresses this with a formalization that:

• Distinguishes direct dependencies (things a model explicitly uses) from indirect ones (things those things use)
• Represents different pipeline roles through operation-centered relationships, so a model used as a data filter is categorized differently from one used as an evaluator
• Resolves artifact identities across names, versions, and repositories

The output is a directed graph you can explore interactively. The live demo includes pre-built graphs for OLMo 3, Nemotron 3 Super, DR-Tulu, SmolLM3, and a full merged graph across all four.

What the graphs actually reveal

This is where ModSleuth becomes genuinely alarming for anyone thinking about compliance or reproducibility. These graphs reveal multi-hop license obligations, train-evaluation coupling, discrepancies between released and training-time artifacts, and documentation inconsistencies that would otherwise be difficult to uncover.

Each of those findings has real consequences:

• Multi-hop license obligations: A permissively licensed model may have been trained on data filtered by a model with a non-commercial license. That restriction can propagate forward, even if nobody documented it.
• Train-evaluation coupling: If the same model family is used both to generate training data and to evaluate the final model, benchmark scores may be artificially inflated. ModSleuth surfaces these circular dependencies automatically.
• Documentation inconsistencies: The artifact a team says they used in a paper is sometimes different from what's actually in the training config. ModSleuth catches these mismatches by cross-referencing multiple artifact types.
• Hidden model roles: Models used as judges, OCR systems, data generators, or quality filters often go undocumented in the main release. They show up in the graph anyway.

Why this matters beyond research

The AI supply chain problem is getting harder to ignore. Software supply chain security has moved beyond the visibility era into the governance era, with the industry transitioning to frameworks that treat AI agents as primary actors in the supply chain. Regulatory frameworks like the EU AI Act are increasingly asking deployers to account for what their models are built on, and that question is now nearly impossible to answer without tooling like this.

There's also a benchmark integrity angle. If a model used as a judge during RLHF training is the same model family used to score the final model on a leaderboard, the evaluation is compromised. ModSleuth makes these relationships visible for the first time at scale.

The practical use cases are broad:

• License compliance: Trace whether any upstream artifact carries a license restriction that propagates to your model
• Benchmark auditing: Identify train-eval coupling that could inflate reported scores
• Regulatory documentation: Generate a full provenance record for AI Act or similar compliance requirements
• Research reproducibility: Verify that the artifacts described in a paper match what was actually used
• Supply chain risk: Understand which upstream models your model inherits behavior from

What it can't do yet

ModSleuth works from public artifacts only. If a dependency isn't documented anywhere publicly, it won't appear in the graph. This is a fundamental constraint: closed or poorly documented models will have incomplete graphs, and the tool is only as good as the documentation ecosystem around the models it analyzes. The researchers acknowledge that the field's documentation practices are inconsistent enough that reconciling artifact identities across releases is itself a major unsolved problem.

It also doesn't yet cover proprietary model releases where training details are not public, which is precisely where supply chain risks tend to be highest.

A new assumption for the field

The release of ModSleuth forces a rethink of a quiet assumption that has been baked into how the field talks about open-source models: that publishing weights and a model card constitutes transparency. The full dependency structure is fragmented across heterogeneous public artifacts, with complexity and recursive depth far outpacing humans' ability to trace. A model card is not a bill of materials. ModSleuth is the first serious attempt to build one automatically.

The paper, authored by Sanjay Adhikesaven, Haoxiang Sun, and Sewon Min at Ai2, is available on arXiv. The tool and resulting dependency graphs are released openly to support transparent analysis of modern LLM ecosystems.