Skip to main content
Back

NovelAPIBench Shows Why Code AI Needs Both RAG and Fine-Tuning

NovelAPIBench reveals that usage examples beat signatures alone, and fine-tuning teaches models to use docs rather than memorize APIs

Paper, Full Access
  • NovelAPIBench is a new automated benchmark that diagnoses why LLMs fail on novel APIs, using six failure categories instead of pass/fail. [Paper]
  • Usage examples are the strongest single knowledge signal; raw source code in context can hurt performance by increasing import errors.
  • Fine-tuning does not replace retrieval for novel APIs; it teaches models to better use retrieved docs, a skill that transfers to unseen libraries.
  • The best two-component knowledge bundle (signature + examples or signature + mechanism) varies by domain and model backbone.
  • Tested across ~1,900 tasks, 4 base models, and 5 domains, comparing RAG, SFT, RAFT, GRACE, and MEMIT adaptation strategies.
  • Retrieval and fine-tuning play complementary roles: retrieval supplies volatile API content, fine-tuning improves procedural integration.

Every time a major Python library ships a new release, the knowledge baked into your favorite code-generation model goes a little stale. A new paper from researchers at Jinnuo Liu, Yue Peng, Jinhan Niu, and Hongyi Wen tackles this head-on by asking a deceptively hard question: when an LLM encounters an API it has never seen before, exactly what kind of knowledge does it need to use it correctly?

The gap no benchmark was measuring

Code LLMs are increasingly used by developers, but these models can generate incorrect API-related code -- known as API hallucinations -- especially when under-trained on certain under-represented APIs. The problem compounds as libraries evolve. Given the dynamic nature of publicly available source-code libraries like PyTorch, which undergo frequent expansion and updates, deprecated calling methods can become a significant challenge. If Code LLMs are not updated in tandem with the latest functions and APIs, this can introduce potential errors and safety risks.

Existing benchmarks for this problem have a fundamental weakness: they are static, use coarse pass/fail scoring, or rely on synthetic APIs that don't reflect how real libraries actually evolve. You can't tell why a model failed, only that it failed. That diagnostic blindspot is exactly what this paper sets out to fix.

What NovelAPIBench actually does

The researchers introduce NovelAPIBench, a fully automated, dynamic benchmark pipeline. Unlike static benchmarks, it can be pointed at any base model and any target library to automatically discover which APIs are genuinely novel to that model, then build a test suite around them. The key concept is the