RiTeK Exposes How Badly AI Retrievers Fail on Medical Knowledge Graphs
A new medical knowledge graph benchmark reveals that top retrieval methods still stumble on multi-hop clinical questions with textual constraints.
ProTakeaways
- RiTeK is a new medical textual knowledge graph benchmark with 15,557 queries across six reasoning topologies.
- Two subsets: RiTeK-PharmKG (drugs, genes, diseases) and RiTeK-ADint (Alzheimer's non-pharmacological interventions).
- Queries mix graph paths with textual constraints on the answer, forcing retrievers to read entity descriptions.
- 11 retrievers tested; prompting methods like ToT/GoT fail, GCR and GNN-RAG lead in supervised settings.
- Even the best models miss rare biomedical links, with GPT-4 hallucinating incorrect diseases in case studies.
- Dataset and code available on GitHub for reproducing results and testing new retrievers.
Medical questions rarely arrive in the clean, single-hop form that most retrieval-augmented systems were built for. A physician asking which organ function in circulating fetal blood is affected by Fetal Distress needs a system that can walk a knowledge graph, respect textual constraints on the answer, and return a specific entity rather than a paragraph of plausible-sounding prose. A new dataset called RiTeK was built to expose how badly current LLM-based retrievers handle that job in the medical domain.
Why medical knowledge graphs keep breaking retrievers
Textual knowledge graphs, or TKGs, mix structured triples (Fetal Distress affects Placental Circulation) with unstructured text descriptions attached to each node. That combination is powerful for medicine, where the same term can carry pages of clinical context, but it also makes retrieval hard. Existing datasets tend to be overly simplistic, typically limited to 1-2 hop reasoning paths, and lack diverse topological structures, rich relation types, or complex constraints. The resulting benchmarks reward shallow lookup and hide the failures that matter in clinical settings.
The RiTeK team argues that STaRK is the only prior dataset integrating relational and textual information for question answering over TKGs, but its topological coverage is limited and its node descriptions are sparse. RiTeK fixes both problems while adding constraints that force retrievers to actually read the text attached to candidate answers.
Two graphs, six reasoning shapes, ten thousand questions
The authors built two medical TKGs on top of existing biomedical resources. RiTeK-PharmKG covers over 500,000 interconnections between genes, drugs, and diseases across 29 relationship types and roughly 8,000 disambiguated entities, enriched with textual descriptions pulled from Ensembl, UMLS, and the Mondo Disease Ontology. RiTeK-ADint is built on ADInt, a knowledge graph focused on non-pharmacological interventions for Alzheimer's disease, with 162,212 entities across 113 UMLS semantic types and 1,017,284 triples covering 15 relation types.