Skip to content

Which literature source builds the best knowledge graph?

For building a knowledge graph with graffold, does a paid semantic-search source (Elicit, Consensus) produce a better, more trustworthy KG than the free PubMed / Europe PMC APIs — and if so, which one is worth paying for?

We answer with one fair experiment: identical queries, identical pipeline, only the source changes. Canonical run: 80 queries across four animal-health target profiles, ~550–590 papers/source, ~4,000 entities each.

Bottom line

Pay for Elicit, and keep the free sources too. Elicit builds the leanest, best-corroborated graph, retrieves an almost entirely different slice of the literature (so it stacks with PubMed/Europe PMC), and its corroboration is the strongest above chance. Consensus points the same way but we could only sample it (free tier caps at 30 searches/month) — it needs a Pro key to rank.

The story in three plots

Elicit's polygon is largest — it leads on corroboration, uniqueness, and above-chance signal, and is the leanest. The free sources are solid but smaller on every axis.

The best KG sits upper-left: fewer entities (less noise) and higher ≥2-source support (more corroboration). Elicit is there — ~15% fewer entities than PubMed with the highest corroboration.

The key result: split the papers in half; a corroborated edge (≥2 papers) recurs in the independent other half ~60% of the time, a single-source edge only 5–7%. Support isn't just a count — it predicts which relationships are real. Holds for every source.

The numbers

Source Papers Entities ≥2-source support Corroboration above chance Recurs in held-out half
PubMed 546 3993 6.4% z=5.5 (p=0.001) 62% vs 5% single
Europe PMC 574 4252 5.3% z=7.3 (p=0.001) 57% vs 5% single
Elicit 593 3428 7.4% z=12.3 (p=0.001) 61% vs 7% single
Consensus* small sample 285* 10.8%* (sample too small) (not measured)

* Consensus row is a small, earlier sample on a different corpus (a 3-query creatine test, before the free-tier 30-search cap was hit) — not scale-comparable to the 80-query rows above. On that sample it reached 10.8% ≥2-source support vs its own run's PubMed 5.1% (~2× corroboration) — the same direction Elicit shows here. Do not read 10.8% as "beats Elicit": the same Europe PMC connector scored 8.3% on that creatine sample but 5.3% here, so cross-corpus numbers are not comparable. A Consensus Pro key would let us run it at this scale for a fair head-to-head.

  • ≥2-source support = fraction of edges backed by a claim in ≥2 distinct papers. This is the KG's trust statistic.
  • Above chance = an agreement-null test (each paper asserts random edges among its own vocabulary); real corroboration beats it at p=0.001 everywhere, strongest for Elicit.
  • Held-out = does a corroborated edge reappear in an independent half of the literature? ~60% do; single-source edges rarely do.
  • Retrieval overlap (title-normalized): PubMed∩Europe PMC 15%; Elicit only 1–3% with either — it is a genuinely different, complementary slice.

See the graphs

Each source's knowledge graph, weighted by corroboration — thick/green edges are backed by multiple papers, thin/grey are single-source. Flip the tabs; Elicit visibly has more green per node.

Graph deferred; small-sample result stands

No weighted graph at this scale — Consensus's free tier (30 searches/mo, resets Oct 1) can't cover an 80-query ingest, and the KG was never built to parquet. But an earlier small-sample run (creatine, ~30 papers) measured the same signature the semantic sources show: ~11% title overlap with PubMed (mostly different, complementary papers) and a denser, more-corroborated KG (285 entities, density 1.63, 10.8% ≥2-source support — ~2× keyword PubMed). A Consensus Pro key fills this tab at scale; the connector is ready (--sources ...,consensus).

Which to pay for, and why it helps graffold

  1. Elicit is proven at scale (two independent runs, same result): leanest graph, highest corroboration, strongest above-chance signal. Its semantic ranking pulls the on-topic, well-witnessed papers that make the trust statistic meaningful.
  2. Keep the free sources. Elicit overlaps them only ~1–3%, so the strongest KG uses both — free for breadth, paid for the corroboration-dense core. Corroboration counts across distinct sources, so more independent sources directly raise the trust signal.
  3. Enrichment is a bonus graffold can use — Elicit/Consensus attach study-type, citation count, and journal quartile per paper, letting graffold weight edges by evidence quality (a heavily-cited RCT vs a lone case report).
  4. Consensus is a plausible alternative but is unranked at scale until run on Pro; on a small sample it behaved like Elicit.

Contradictions are a feature, not noise: contradicted edges carry ~3× the support of ordinary ones because genuine scientific disputes are the heavily-studied relationships — graffold surfaces them with per-side provenance via /v1/kg/contradictions rather than averaging them away.

Reproduce

# 1. cache retrieval per source (free; no LLM)
ELICIT_API_KEY=... NCBI_API_KEY=... python benchmarks/source_eval.py \
  --queries queries.txt --sources pubmed,europepmc,elicit --limit 10 --out /tmp/eval
# 2. extract + null tests + weighted graph viz (Bedrock Haiku)
AWS_PROFILE=default AWS_REGION=us-east-1 python benchmarks/source_eval.py \
  --queries queries.txt --sources pubmed,europepmc,elicit --limit 10 \
  --extract --null --viz --service bedrock \
  --model global.anthropic.claude-haiku-4-5-20251001-v1:0 --out /tmp/eval
# 3. summary plots (radar / scatter / recurrence)
python benchmarks/summary_plots.py /tmp/eval/results.json /tmp/eval/summary

Elicit needs a Pro-plan ELICIT_API_KEY; Consensus a CONSENSUS_API_KEY (free tier 30 searches/mo). Free sources need no key. Cached docs_*.json let retrieval and extraction run as separate steps.

Scope

Single domain (animal-health target profiles), single run, Haiku extraction, Consensus not scaled (quota). Directional, not definitive — the relative ordering (Elicit leads; all sources beat chance) is the finding. Re-run on your own queries before a purchasing decision.