Skip to content

graffold-ingest

Domain-agnostic knowledge graph ingestion. Extracts entities and relationships from any source, resolves them to canonical IDs, and publishes to your graph database of choice.

Quick Start

# Install (Python 3.12+)
uv sync --extra all             # from a clone — core + LLM + graph + storage
# or the standalone binary (no Python needed): curl -fsSL https://github.com/graffold/graffold-ingest-releases/releases/latest/download/install.sh | sh

# One-time setup (creates ~/.graffold/config.toml, validates API key)
graffold-ingest init --api-key sk-ant-... --atlas-dir ~/atlas/programs/

# Verify config, backend connectivity, and store
graffold-ingest status

# Build a KG from Atlas output (incremental, config-aware)
graffold-ingest watch

From install to a running knowledge graph.

After setup

# Ingest a program directory (one-shot)
graffold-ingest ingest ~/atlas/programs/etec-pigs/v1/

# Pull literature straight from Europe PMC / PubMed
graffold-ingest pipeline --source europepmc \
    --query "F18 ETEC adhesion piglet" --full-text --limit 50 \
    --service bedrock-llama

# Ask the graph a question
graffold-ingest ask "What feed-additive candidates target F18 adhesion?"

# Generate prior-knowledge.md for Atlas's next run
graffold-ingest context "post-weaning ETEC" -d ~/atlas/programs/etec-v2/v1/

Architecture

flowchart LR
 subgraph Sources
 W[Web/URL]
 P[PDF]
 C[CSV/Excel]
 D[Database]
 A[External pipeline]
 end

 subgraph Pipeline
 direction TB
 F[Fetch] --> CH[Chunk]
 CH --> EX[Extract]
 EX --> RE[Resolve]
 RE --> EM[Embed]
 EM --> PU[Publish]
 RE --> CO[Community]
 end

 subgraph Storage
 PQ[(Parquet)]
 N4[(Neo4j)]
 NP[(Neptune)]
 DK[(DuckDB)]
 end

 subgraph Query
 DR[DRIFT]
 GL[Global]
 LO[Local]
 end

 W --> F
 P --> F
 C --> F
 D --> F
 A --> F
 PU --> PQ
 PU --> N4
 PU --> NP
 PU --> DK
 PQ --> DR
 PQ --> GL
 PQ --> LO

Design Decisions

Schema-free by default, schema-guided when needed

graffold-ingest ships with a generic schema (default_schema.yaml) that handles Person, Organization, Product, Concept, etc. For domain-specific work, provide a custom schema:

graffold-ingest pipeline --source web --url "..." --schema-path my-domain.yaml

Or discover one automatically from sample data:

graffold-ingest discover --domain "veterinary drug discovery" --from-file brief.md -o schema.yaml

Parquet as source of truth

All extracted data lands in Parquet first. Graph databases (Neo4j, Neptune) are downstream consumers — you can rebuild them from Parquet at any time. This gives you:

  • Version control (git-friendly columnar files)
  • Portability (no vendor lock-in)
  • Analytics (DuckDB queries over the raw data)
  • Reproducibility (re-resolve, re-embed without re-extracting)

Incremental by default

Re-running the pipeline on the same source skips already-ingested documents. Use --no-incremental to force re-extraction.

Multi-tenant

The API layer supports multiple tenants via API keys. Each tenant's data is isolated by project_id.

Connectors

Source Class What it fetches
web WebConnector URLs, sitemaps, recursive crawl (depth-limited)
pdf PDFConnector Local PDFs via pymupdf/pdfplumber
api APIConnector REST APIs with pagination, auth headers
csv CSVConnector CSV, TSV, Excel, Parquet tabular files
database DatabaseConnector SQL queries via connection string
agteria AgteriaConnector Atlas pipeline filesystem (phase-*.md)
pubmed PubMedConnector PubMed abstracts (rate-limited 3/s, 10/s with API key)
europepmc EuropePMCConnector Europe PMC full-text (Open Access, ~20K chars/paper) + abstract fallback

Literature ingestion

The PubMed and Europe PMC connectors turn published research into graph entities. Europe PMC is preferred for Open Access papers (full text, ~20K chars) with PubMed abstract fallback for non-OA:

# Ingest literature on a topic
graffold-ingest pipeline --source europepmc --url "ETEC F18 adhesin" --service bedrock-llama

Full-text extraction chunks large papers, applies a relevance gate (drops off-topic contamination), and is checkpointed so long runs resume after interruption.

Writing a custom connector

from graffold_ingest.connectors.base import BaseConnector, Document

class MyConnector(BaseConnector):
 async def fetch(self) -> list[Document]:
 return [Document(id="doc-1", content="...", source_type="custom")]

Pipeline Stages

Chunk

Splits documents into LLM-sized pieces. Preserves sentence boundaries.

Parameter Default Description
chunk_size 2000 chars Target chunk size
overlap 200 chars Overlap between chunks

Extract

LLM reads each chunk and extracts structured entities + relationships.

  • Schema-free: generic extraction (Person, Organization, Concept, etc.)
  • Schema-guided: domain-specific types (Target, Disease, Compound, etc.)
  • Supports: Anthropic, Bedrock, OpenAI, Ollama, Cloudflare Workers AI

Resolve

Deduplicates and canonicalizes entities:

  1. Case normalization — tp53 → TP53
  2. Alias resolution — RANTES → CCL5 (via HGNC synonyms)
  3. Authority lookup — UniProt, MONDO, PubChem REST APIs
  4. Fuzzy matching — optional Levenshtein-based dedup

Community Detection

Leiden clustering at multiple resolution levels. Enables:

  • Community-level summaries for Global search
  • Hierarchical navigation (corpus → domain → community → entity)
  • Hub identification

Publish

MERGE to graph database with full provenance (source_doc_id, extraction_method, ingested_at).

Embed

Generate vector embeddings for semantic search:

  • Cloudflare Workers AI (remote, free tier)
  • nomic-embed-text via Ollama (local)
  • Any OpenAI-compatible endpoint

Entity Resolution

Resolver Entities Authority Method
UniProt Protein, Target, Enzyme UniProt REST ID mapping + synonym table
MONDO Disease, Condition EBI OLS4 Ontology traversal
PubChem Compound, Drug, Chemical PubChem PUG Name → CID resolution
Local All types HGNC aliases Case norm + synonym + optional fuzzy
Composite All All of the above Cascading resolution

Search Modes

Mode Algorithm When to use
DRIFT Multi-hop iterative reasoning over graph structure "What pathway connects X to Y?"
Global Map-reduce over Leiden community summaries "What are the main themes in this corpus?"
Local Vector similarity + text match "What is CpTrxR?"
Auto Selects based on question structure Default — works for most queries

Storage Backends

Backend Protocol Best for
neo4j Bolt Default — works with Neo4j and Memgraph
neptune OpenCypher + IAM SigV4 AWS production
duckdb SQL over Parquet Local analytics, zero infrastructure
parquet File I/O Source of truth, version control

Integration Patterns

With an external pipeline (filesystem watch)

# Config-aware watch — reads atlas_programs_dir from ~/.graffold/config.toml
graffold-ingest watch

# Or specify explicitly
graffold-ingest watch ~/atlas/programs/ --poll 60

The pipeline writes phase files to an output directory; graffold-ingest watches it and ingests each new phase file as it appears — no pipeline code changes needed. Incremental by default: fingerprints files so re-running only processes new/changed phase files.

One-shot ingest

# Ingest a single program directory (uses config defaults)
graffold-ingest ingest ~/atlas/programs/crypto-v11/

Simpler than pipeline --source agteria --path ... — one command, uses config defaults for LLM service, chunk size, and output.

With an external platform (HTTP push)

# Temporal workflow activity
await httpx_client.post(f"{GRAFFOLD_INGEST_URL}/v1/entities", json={
 "source_run_id": "run-42",
 "source_system": "external",
 "entities": [...],
 "relationships": [...],
})

With graffold-api (query layer)

graffold-ingest publishes to the same graph database that graffold-api queries. The frontend at app.graffold.com sends queries to graffold-api, which reads from Neo4j/Memgraph.

Feedback loop

# Generate prior-knowledge.md directly into the program directory
graffold-ingest context cryptosporidiosis

# Or ask a specific question
graffold-ingest ask "What targets have been killed and why?"

context outputs prior-knowledge.md into the program dir — a file Atlas consumes at startup for cross-run memory. ask is a general natural language query over the graph via DRIFT search.

Configuration

# LLM
GRAFFOLD_LLM_SERVICE=anthropic # bedrock | bedrock-llama | anthropic | openai | ollama | cloudflare | openrouter
GRAFFOLD_LLM_MODEL=claude-haiku-4-5-20251001
# bedrock-llama uses us.meta.llama3-3-70b-instruct-v1:0 — ~5x cheaper than Claude on output
NCBI_API_KEY=... # Faster PubMed (10/s vs 3/s)

# Graph
GRAPH_BACKEND=neo4j # neo4j | neptune | duckdb
NEO4J_URI=bolt://localhost:7687
NEO4J_USER=neo4j
NEO4J_PASSWORD=secret

# Resolution
GRAFFOLD_RESOLVE=true
NCBI_API_KEY=... # Optional — faster PubMed

# Storage
PARQUET_DIR=~/.graffold/parquet

# Multi-tenant API
TENANT_KEYS=sk-abc:tenant1,sk-xyz:tenant2
INGEST_INTERNAL_MODE=false # Require auth

Benchmarks

Operation Throughput Notes
Structured ingest 180,000 entities/sec CSV → Parquet
Tabular chunking 920,000 rows/sec CSV/TSV → markdown chunks
Entity resolution 800,000 entities/sec HGNC + synonym (no fuzzy)
Entity resolution F1 = 1.00 9/9 true merges, 0 false merges
LLM extraction ~6 entities/sec Ollama qwen3:1.7b (local)
Query (local LLM) 3–4 sec Discovery → Expand → Synthesize
Storage ~27 bytes/entity Parquet (columnar, compressed)

See Atlas Pipeline Benchmark for a full real-world run.