Skip to main content
Back

sift-kg Turns Any Folder of PDFs Into an Interactive Knowledge Graph

A CLI tool that converts document collections into browsable knowledge graphs, extracting entities and relations via LLM with human-approved deduplication.

Repo, Full Access
  • sift-kg is a Python CLI that turns document folders into interactive knowledge graphs
  • Supports 75+ formats via Kreuzberg, with local OCR through Tesseract, EasyOCR, or PaddleOCR
  • Schema-free mode has the LLM design entity and relation types tailored to your corpus
  • Human-in-the-loop deduplication combines SemHash pre-dedup, LLM proposals, and YAML review
  • Works with OpenAI, Anthropic, Mistral, Ollama, or any LiteLLM provider, with cost caps
  • Exports to GraphML, GEXF, SQLite, CSV, and JSON for agents or analysis tools

Building a personal knowledge base has become a rite of passage, and also a black hole for weekends. A new open-source CLI called sift-kg proposes a shortcut: point it at a folder of PDFs, papers, or records, and it hands back an interactive knowledge graph showing how everything connects. No database, no infrastructure, no manual tagging.

The pipeline runs entities and relations through an LLM, deduplicates them with your approval, and generates a browser-based viewer you can explore. The same structured JSON that powers the visualization can be fed back into AI agents as persistent memory, turning a one-off analysis into something that compounds over time.

From folder to graph in one pipeline

The tool is a Python package installed with pip install sift-kg and driven by a small set of subcommands. A typical run looks like this:

sift init                  # create sift.yaml + .env.example
sift extract ./documents/  # pull entities & relations
sift build                 # assemble the knowledge graph
sift resolve               # find duplicate entities
sift review                # approve/reject merges
sift apply-merges          # commit your decisions
sift view                  # open interactive graph in browser

Text extraction is handled locally by Kreuzberg with optional OCR through Tesseract, EasyOCR, PaddleOCR, or Google Cloud Vision. An LLM then either uses a predefined domain schema or designs entity and relation types from the corpus itself, extracts them, builds a NetworkX graph, proposes entity resolutions for human review, and finally generates a narrative and an interactive viewer. Every entity and relation links back to the source document and passage, which matters if you ever need to defend a claim.