Skip to content

Graffold Ingest

Domain-agnostic ingestion agent that turns anything into a knowledge graph — web pages, PDFs, APIs, CSVs, databases.

Separate Repository

graffold-ingest lives at github.com/graffold/graffold-ingest. This documentation is pulled into the unified wiki at build time.

Quick Start

uv sync
graffold-ingest tui        # Interactive terminal UI
graffold-ingest scrape     # Scrape a URL or file
graffold-ingest pipeline   # Run full ingestion pipeline

Architecture

┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│  Connectors │───▸│   Chunker   │───▸│  Extractor  │───▸│  Publisher  │
│ web/pdf/api │    │  split docs │    │ LLM entities│    │ graph write │
└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘
                                            │
                                     ┌──────┴──────┐
                                     │  Resolver   │
                                     │ dedup/merge │
                                     └─────────────┘

Connectors

Connector Sources Description
web Any URL, sitemaps, crawling HTML → Markdown extraction
pdf Local PDFs MarkItDown / Docling extraction
api REST APIs with pagination JSON → structured documents
csv CSV, Excel, Parquet files Tabular → entity/relationship mapping
database SQL databases Query → document extraction

Science Connectors

Connector Data Source What It Fetches
openalex OpenAlex Literature, citations, abstract reconstruction
pubmed NCBI PubMed Abstracts via E-utilities (search, summary)
opentargets Open Targets Target-disease associations, drugs, tractability
string STRING DB Protein-protein interactions, functional enrichment

Pipeline

graffold-ingest pipeline \
  --source web \
  --url "https://example.com/docs" \
  --database memgraph \
  --service bedrock

Pipeline Stages

  1. Connect — Fetch raw content from source
  2. Chunk — Split into token-bounded pieces with overlap
  3. Extract — LLM entity/relationship extraction (with gleaning)
  4. Resolve — Deduplicate entities (ID match → synonym → fuzzy)
  5. Publish — Write to graph database (Neo4j, Memgraph, FalkorDB)

Adapter Layer

graffold-ingest includes a compatibility layer for platform integrations:

Adapter Purpose
SearchClient Wire-compatible search protocol for any upstream
ClientPool Cached drop-in replacements (OpenAlex, EuropePMC, PubChem, PubMed, UniProt)
KGQueryClient Protocol for querying existing KG data

Configuration

# .env
LLM_SERVICE=bedrock
BEDROCK_MODEL_ID=us.meta.llama3-1-8b-instruct-v1:0
NEO4J_URI=bolt://localhost:7687
NEO4J_PASSWORD=changeme

Relationship to graffold-api

graffold-ingest is the standalone ingestion engine. graffold-api uses it (or its own pipeline/ modules) for API-driven ingestion jobs. They share:

  • The same graph schema (entity types, relationship types)
  • The same database backends (Neo4j, Memgraph, FalkorDB)
  • Compatible chunking and extraction patterns

The split allows graffold-ingest to run as a CLI/agent independently of the API server.