Ingestion Benchmarks

Performance benchmarks for the graffold-ingest pipeline, measuring throughput, entity resolution accuracy, and end-to-end demo results.

Running Benchmarks

cd graffold-ingest
uv sync --extra all
python benchmarks/ingest_throughput.py
python benchmarks/resolution_accuracy.py
python benchmarks/full_demo.py  # requires Ollama + qwen3:1.7b

---

Ingest Throughput

Measures entities/second for structured data ingestion (CSV → resolve → Parquet).

Workload	Entities	Time	Throughput
Structured CSV (5,000 entities)	5,000	~0.8s	~6,000 entities/sec
Structured CSV (20,000 entities)	20,000	~3.2s	~6,200 entities/sec
Tabular chunking (10,000 rows)	10,000	~0.3s	~33,000 rows/sec
Tabular chunking (50,000 rows)	50,000	~1.4s	~35,000 rows/sec

Source

"""Benchmark: ingestion throughput (entities/sec)."""

async def bench_structured_ingest(n_entities: int = 5000) -> dict:
    """CSV → resolve → publish to Parquet."""
    # Generate sample proteomics CSV
    # Parse → build nodes/edges → resolve → write Parquet
    # Return {entities, relationships, seconds, entities_per_sec}

async def bench_tabular_chunking(n_rows: int = 10000) -> dict:
    """Measure tabular chunking speed (rows → chunks)."""
    # chunk_tabular(csv_path, rows_per_chunk=50)
    # Return {rows, chunks, seconds, rows_per_sec}

---

Entity Resolution Accuracy

Measures precision/recall of synonym-based entity merging.

Ground Truth Test Set

Should merge (synonyms):

Name A	Name B	Type
RANTES	CCL5	Chemokine synonym
MCP-1	CCL2	Chemokine synonym
eotaxin	CCL11	Chemokine synonym
FGF-basic	FGF2	Growth factor alias
FGF (basic)	FGF2	Parenthetical variant
IP-10	CXCL10	Chemokine synonym
SDF-1	CXCL12	Chemokine synonym
tp53	TP53	Case normalization
gfap	GFAP	Case normalization

Should NOT merge (distinct entities):

Name A	Name B	Reason
TP53	BRCA1	Different genes
IL6	IL8	Different interleukins
VEGFA	VEGFB	Different isoforms
CCL5	CXCL10	Different chemokines
JAK1	JAK2	Different kinases

Results

Metric	Score
Precision	1.000
Recall	0.778
F1	0.875
True Positives	7/9
False Negatives	2/9
True Negatives	5/5
False Positives	0/5

Missed merges

The resolver uses a built-in synonym dictionary. Misses occur for aliases not yet in the dictionary (e.g., less common chemokine names). Adding more synonyms improves recall without affecting precision.

Resolution Speed

Workload	Input	Output	Merged	Time	Throughput
10,000 entities	11,000	~10,000	~1,000	~0.4s	~27,000/sec
50,000 entities	55,000	~50,000	~5,000	~2.1s	~26,000/sec

---

Full End-to-End Demo

Complete pipeline: structured data + LLM extraction + query.

Prerequisites

• Ollama running with qwen3:1.7b and nomic-embed-text
• graffold-ingest installed with --extra all

Pipeline

Run 1: Structured ingest (10 proteins from sample data)
  → CSV → nodes/edges → resolve → Parquet
  → ~10 entities, ~10 relationships (<1s)

Run 2: LLM extraction (3 PubMed abstracts)
  → Abstracts → Ollama qwen3:1.7b → entities/edges → resolve → Parquet
  → ~15-25 entities, ~10-20 relationships (~5-15s depending on hardware)

Query: DuckDB backend + Ollama RAG
  → "What drugs target VEGFA?" → Bevacizumab, Faricimab
  → "What is tau protein associated with?" → Alzheimer's, neurodegeneration
  → "Which proteins are on the Inflammation panel?" → IL6, TNF, CXCL8

Sample Output

══════════════════════════════════════════════════════════════════════
  GRAFFOLD DEMO — Knowledge Graph Builder + Query Agent
══════════════════════════════════════════════════════════════════════

▶ Run 1: Structured ingest (sample proteomics data)
  ✓ 13 entities, 10 relationships (0.4s)

▶ Run 2: LLM extraction (3 PubMed abstracts via Ollama)
  ✓ 22 entities, 18 relationships (8.3s)

▶ Querying the graph...

  Q: What drugs target VEGFA?
  A: Bevacizumab inhibits VEGFA. Faricimab dual-targets VEGFA and ANGPT2...
     (1.2s, 3 entities found)

  Q: What is tau protein associated with?
  A: Tau protein aggregation is a hallmark of Alzheimer disease...
     (1.0s, 4 entities found)

══════════════════════════════════════════════════════════════════════
  FINAL GRAPH: 35 entities, 28 relationships
  Storage: 12 KB
══════════════════════════════════════════════════════════════════════

---

Performance Summary

Stage	Throughput	Notes
CSV parsing + publish	~6,000 entities/sec	Pure Python, no LLM
Tabular chunking	~35,000 rows/sec	Memory-efficient
Entity resolution	~27,000 entities/sec	Synonym dictionary lookup
LLM extraction	~2-5 entities/sec	Bottleneck: Ollama inference
Query (RAG)	~1-2s per question	DuckDB + Ollama

Scaling LLM extraction

LLM extraction is the bottleneck. To scale:

• Use Bedrock (faster inference than local Ollama)
• Increase chunk_size to reduce LLM calls
• Run multiple workers via InMemoryJobQueue(max_workers=4)
• For structured data, bypass LLM entirely (direct CSV → graph)

---

External Pipeline Integration Benchmarks

When using graffold-ingest as the background worker for an external pipeline:

Pattern	Latency	Notes
Entity push (<50 entities)	<100ms	Inline, no queue
Entity push (≥50 entities)	<200ms (accepted)	Async job, returns immediately
Full pipeline (web scrape)	30-120s	Depends on content size
Full pipeline (PDF)	10-60s	Depends on page count
Coverage check (graffold-api)	<50ms	Direct Neo4j/DuckDB query
Retrieve (graffold-api)	<8s	Vector + graph traversal