Zed Beats 25 AI Search Methods With Simple Weighted BM25 for MCP Tools

Zed selected weighted BM25 for MCP tool search after comparing 25 approaches, including decision models, LLM routers, and dense embeddings. According to Zed’s evaluation, the chosen system runs locally without model inference or API calls, giving the editor a fast way to search hundreds of tools without placing every definition in the model’s context.

The Model Context Protocol, or MCP, lets servers expose callable tools to an AI agent. Each tool usually includes a name, description, input schema, and server metadata. Sending every definition to the model consumes tokens and can make similar tools harder to distinguish. Restricting the set too aggressively can hide the capability needed for a task.

Inside Zed’s 25-way test

Zed’s selection reflects the constraints of a local-first editor: low latency, offline operation, predictable behavior, and minimal runtime dependencies. The result offers a practical production trade-off rather than a universal ranking of retrieval methods across every catalog and workload.

Why BM25 likes tool catalogs

BM25 ranks documents by matching query terms against indexed text. Its scoring combines three signals:

• Term frequency: repeated matches increase the score with diminishing returns.
• Inverse document frequency: rare terms receive more weight than common terms.
• Length normalization: longer entries receive an adjustment so they do not rank highly through size alone.

Implementations commonly tune k1, which controls term-frequency saturation, and b, which controls length normalization. Field weighting can then assign different importance to tool names, descriptions, parameter names, and server namespaces.

Tool metadata suits lexical retrieval because it is short, structured, and rich in distinctive identifiers such as github, postgres, and jira. BM25 also exposes the terms and field scores behind each result, which helps developers investigate incorrect routing.

Weighting does the hard work

Standard BM25 can overvalue common action verbs. A query such as “Create a Jira ticket” may reward unrelated tools containing create, even when jira provides the stronger clue. As verbs such as create, get, and list spread across a catalog, their value for distinguishing tools falls.

Separate benchmark results illustrate the effect of tuning. In StackOne’s 2,700-query evaluation, standard BM25 reached 13.7% top-1 accuracy, while a TF-IDF-adjusted variant reached 21.3%. That is a gain of 7.6 percentage points, or about 55% relative. Top-1 accuracy measures how often the expected tool occupies the first result. StackOne and Zed used different query sets and catalogs, so their absolute scores are not directly comparable.

Common refinements for tool catalogs include:

• Giving tool names and product identifiers more weight than prose descriptions
• Reducing the influence of frequent action verbs while retaining them as operation signals
• Splitting snake_case, kebab-case, and camelCase identifiers into searchable terms
• Expanding known abbreviations and aliases
• Boosting server or namespace matches when the query names a specific integration

Zed describes its implementation as weighted BM25 with a small set of normalization rules. Exact weights remain catalog-dependent and should be measured against representative queries rather than copied as universal constants.

The trade-offs behind local search

StackOne’s broader comparison placed embedding retrieval near 38% top-1 accuracy and rerankers above 40%, with reranking latency around 200 to 500 milliseconds. Those figures come from a separate evaluation, but they show the available trade-off: semantic methods can improve matching while adding models, latency, and deployment work.

A two-stage MCP pattern

Anthropic’s tool-search feature also uses a BM25 variant, while FastMCP provides a BM25SearchTransform. FastMCP builds its in-memory index on the first search and rebuilds it when the catalog changes.

Both implementations support a two-stage retrieval pattern:

1. Expose a compact search tool instead of every complete tool schema.
2. Search names, descriptions, parameters, and server metadata.
3. Return a small candidate set based on the query.
4. Load complete schemas only for the selected candidates.

This pattern reduces context use while preserving access to a large catalog. The appropriate candidate count depends on retrieval accuracy, schema size, and the model’s ability to choose among similar results.

Catalog size is only one pressure

Vocabulary overlap determines retrieval difficulty more directly than the raw number of tools. A catalog containing thousands of clearly named, distinct operations can remain searchable with BM25, while a smaller catalog filled with near-duplicate names and generic descriptions can produce weak rankings.

Semantic retrieval becomes more useful when users rely on synonyms absent from tool metadata, descriptions are multilingual, or several integrations expose nearly identical operations. A hybrid design can use BM25 for candidate generation, add embedding results for semantic coverage, and reserve a cross-encoder or LLM for reranking a short list.

A rollout checklist

• Index tool names, descriptions, parameter names, and server namespaces as separate fields.
• Normalize identifier formats and maintain aliases for common product abbreviations.
• Filter tools by permissions and availability before ranking them.
• Build an evaluation set from real requests, including vague phrasing, abbreviations, and misspellings.
• Measure top-1 accuracy, recall at the candidate cutoff, latency, and index-rebuild time.
• Inspect failed queries before adding semantic retrieval or model-based reranking.

For a local editor with a moderately sized MCP catalog, weighted BM25 provides fast retrieval, transparent debugging, and offline execution. Zed’s evaluation supplies a concrete default, while catalog-specific tests determine when additional semantic layers earn their cost.