LangWatch Unifies Open-Source LLM Tracing and Evals in One Platform

Shipping LLM agents into production still feels like flying blind. Traces live in one tool, evals in another, prompts in a third, and none of them agree on what a failure looks like. LangWatch is trying to collapse that stack into a single open-source platform, and the repo has now crossed 3.5k stars while adding an AI gateway, agent simulations, and a one-command local install.

At its core, LangWatch is a testing and observability platform for LLM applications. It traces production traffic, capturing prompts, completions, tool calls, latency, tokens, and cost, then evaluates quality offline against a curated dataset and online against a sample of live traffic. Trace, dataset, evaluate, and optimize prompts or models all live in one loop, without glue code or tool sprawl.

The tracing and eval loop

The mental model is standard OpenTelemetry. You instrument your application, typically with a decorator or SDK wrapper, so each LLM interaction becomes a trace, which is a tree of spans representing the model calls, retrieval steps, and tool invocations. Each span records inputs, outputs, latency, token usage, and cost. Because the platform is OTLP-native, it accepts data from any OpenTelemetry-compatible library out of the box, which avoids vendor SDK lock-in.