---
title: "NovelAPIBench Shows Why Code AI Needs Both RAG and Fine-Tuning"
subtitle: "NovelAPIBench reveals that usage examples beat signatures alone, and fine-tuning teaches models to use docs rather than memorize APIs"
url: "https://alphasignal.ai/news/novelapibench-shows-why-code-ai-needs-both-rag-and-fine-tuning"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-06-02T00:00:00.000Z"
updated: "2026-06-10T07:45:56.313Z"
type: "Paper"
topics:
  - "Agents"
  - "Development"
subtopics:
  - "Tool Use"
  - "Code Agents"
  - "Code Agents"
  - "Code Generation"
upvotes: 1232
source_url: "https://arxiv.org/abs/2606.03657"
comments: 0
truncated: true
---

# NovelAPIBench Shows Why Code AI Needs Both RAG and Fine-Tuning

> NovelAPIBench reveals that usage examples beat signatures alone, and fine-tuning teaches models to use docs rather than memorize APIs

Every time a major Python library ships a new release, the knowledge baked into your favorite code-generation model goes a little stale. [A new paper](https://arxiv.org/abs/2606.03657) from researchers at Jinnuo Liu, Yue Peng, Jinhan Niu, and Hongyi Wen tackles this head-on by asking a deceptively hard question: when an LLM encounters an API it has never seen before, exactly what kind of knowledge does it need to use it correctly?

## The gap no benchmark was measuring

Code LLMs are increasingly used by developers, but these models can generate incorrect API-related code -- known as API hallucinations -- especially when under-trained on certain under-represented APIs. The problem compounds as libraries evolve. Given the dynamic nature of publicly available source-code libraries like PyTorch, which undergo frequent expansion and updates, deprecated calling methods can become a significant challenge. If Code LLMs are not updated in tandem with the latest functions and APIs, this can introduce potential errors and safety risks.

Existing benchmarks for this problem have a fundamental weakness: they are static, use coarse pass/fail scoring, or rely on synthetic APIs that don't reflect how real libraries actually evolve. You can't tell _why_ a model failed, only _that_ it failed. That diagnostic blindspot is exactly what this paper sets out to fix.

## What NovelAPIBench actually does

The researchers introduce **NovelAPIBench**, a fully automated, dynamic benchmark pipeline. Unlike static benchmarks, it can be pointed at any base model and any target library to automatically discover which APIs are genuinely novel to that model, then build a test suite around them. The key concept is the

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Diagnosing Knowledge Gaps in LLM Tool Use: An Agentic Benchmark for Novel API Acquisition](https://arxiv.org/abs/2606.03657)
- [On Mitigating Code LLM Hallucinations with API Documentation](https://arxiv.org/html/2407.09726)
- [Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs](https://arxiv.org/abs/2312.05934)
- [RAFT: Retrieval Augmented Fine-Tuning](https://techcommunity.microsoft.com/blog/azure-ai-foundry-blog/raft-a-new-way-to-teach-llms-to-be-better-at-rag/4084674)
- [LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation](https://arxiv.org/html/2409.20550v1)