Skip to content

Which literature source builds the best knowledge graph?

For building a knowledge graph with graffold, does a paid semantic-search source (Elicit, Consensus) produce a better, more trustworthy KG than the free PubMed / Europe PMC APIs — and if so, which one is worth paying for?

We answer with one fair experiment: identical queries, identical pipeline, only the source changes. Canonical run: 80 queries across four animal-health target profiles, ~550–700 papers/source, ~3,500–4,600 entities each — now including a full-scale Consensus arm.

Bottom line

The two paid semantic sources (Elicit, Consensus) both win — and they win differently. Both retrieve an almost entirely different slice of the literature (so they stack with the free APIs), and both post the strongest corroboration-above-chance (Consensus z=13.1, Elicit z=12.8 — vs 4–8 for the free sources). Elicit builds the leanest graph (fewest entities, highest ≥2-source rate at 6.8%); Consensus retrieves the most and recalls the most gold (700 papers, 17/29 named targets). Keep the free sources for breadth; add a paid semantic source for the corroboration-dense core.

The story in three plots

Elicit's polygon is largest — it leads on corroboration density and uniqueness and is the leanest. Consensus is close behind with the strongest above-chance signal and widest reach. The free sources are solid but smaller on every axis.

The best KG sits upper-left: fewer entities (less noise) and higher ≥2-source support (more corroboration). Elicit is there — ~15% fewer entities than PubMed with the highest corroboration.

The key result: split the papers in half; a corroborated edge (≥2 papers) recurs in the independent other half ~60% of the time, a single-source edge only 5–7%. Support isn't just a count — it predicts which relationships are real. Holds for every source.

The numbers

80 queries, 4 animal-health target profiles, Bedrock Claude Haiku extraction. All four sources run at full scale.

What is ≥2-source support?

The fraction of edges (relationships) in the KG that are asserted by claims from at least 2 distinct source papers.

Concretely, for each edge (e.g. creatine — increases → muscle_mass), count how many distinct PMIDs (or paper IDs) contain a claim producing that edge. Then:

≥2-source support = (edges with ≥2 distinct papers) / (all edges)

Why it matters: it's the KG's trust statistic. A single paper claiming X causes Y is a lead; two independent papers claiming the same thing is a corroborated claim. The higher this rate, the more of your graph is backed by more than one witness.

In this study: Elicit's KG had 6.8% ≥2-source edges — highest of the four. That means for every 100 edges Elicit's KG builds, ~7 are corroborated by ≥2 papers vs ~5 for Europe PMC. Combined with its lean size (fewest entities), Elicit gives the highest density of corroborated signal per entity.

The null test (next column) then checks: is 6.8% actually above chance, or would you get that just from popular entities co-occurring? Answer for Elicit (z=12.8, p=0.001): well above chance — the corroboration is real, not an artifact of degree.

Source Papers Entities ≥2-source support Corroboration above chance Held-out recurrence Gold recall
PubMed 546 3977 6.1% z=7.7 (p=0.001) 58% vs 5% single 13/29
Europe PMC 574 4120 5.0% z=4.1 (p=0.001) 61% vs 4% single 13/29
Elicit 593 3569 6.8% z=12.8 (p=0.001) 59% vs 7% single 16/29
Consensus 700 4637 6.3% z=13.1 (p=0.001) 57% vs 6% single 17/29
  • ≥2-source support = fraction of edges backed by a claim in ≥2 distinct papers. This is the KG's trust statistic. Elicit leads (6.8%) — leanest graph, densest corroboration per entity.
  • Above chance = an agreement-null test (each paper asserts random edges among its own vocabulary); real corroboration beats it at p=0.001 everywhere, strongest for the two paid semantic sources (Consensus z=13.1, Elicit z=12.8) vs the free APIs (z=4–8).
  • Held-out = does a corroborated edge reappear in an independent half of the literature? ~57–61% do; single-source edges only 4–7%.
  • Gold recall = of 29 named target-profile entities, how many the source recovered from literature alone, unseeded. Consensus 17, Elicit 16 — the semantic sources beat the free APIs (both 13).
  • Retrieval overlap (title-normalized): PubMed∩Europe PMC 15%; the semantic sources are 1–7% with the free APIs — genuinely different, complementary slices. Elicit∩Consensus overlap only 13%, so even the two paid sources largely complement each other.

See the graphs

Each source's knowledge graph, weighted by corroboration — thick/green edges are backed by multiple papers, thin/grey are single-source. Flip the tabs; Elicit visibly has more green per node.

Which to pay for, and why it helps graffold

  1. Both paid semantic sources beat the free APIs on the signal that matters — corroboration above chance (Consensus z=13.1, Elicit z=12.8 vs z=4–8) and gold recall (17, 16 vs 13). Their semantic ranking pulls the on-topic, well-witnessed papers that make the trust statistic meaningful.
  2. Elicit for the leanest, densest graph; Consensus for reach. Elicit builds the fewest entities with the highest ≥2-source rate (6.8%) — less noise per corroborated edge. Consensus retrieves the most papers (700) and recalls the most gold (17/29). They overlap only ~13% with each other, so running both is additive, not redundant.
  3. Keep the free sources. The semantic sources overlap PubMed/Europe PMC only ~1–7%, so the strongest KG uses all of them — free for breadth, paid for the corroboration-dense core. Corroboration counts across distinct sources, so more independent sources directly raise the trust signal.
  4. Enrichment is a bonus graffold can use — Elicit/Consensus attach study-type, citation count, and journal quartile per paper, letting graffold weight edges by evidence quality (a heavily-cited RCT vs a lone case report).

Contradictions are a feature, not noise: contradicted edges carry ~3× the support of ordinary ones because genuine scientific disputes are the heavily-studied relationships — graffold surfaces them with per-side provenance via /v1/kg/contradictions rather than averaging them away.

What Elicit and Consensus are actually selling

A fair question after seeing the numbers: what are you paying for? Both Elicit and Consensus are LLM curation layers on top of the same corpus everyone else has (Semantic Scholar, ~200M papers). They don't sell access — they sell semantic ranking, per-paper enrichment, and claim-level summarization.

Elicit Consensus
Underlying corpus Semantic Scholar (~200M) Semantic Scholar (~200M)
Semantic ranking Embeddings + query rewriting, tuned for research questions Same, tuned for scientific claims
Per-paper LLM extraction Population / intervention / outcome / n / effect direction ("columns") Yes / No / Nuanced verdict on the query claim
Metadata enrichment Study type, citation count, journal quartile Same + "consensus meter" aggregation
Content returned to API Abstracts + extracted fields Abstracts + verdict + snippet
Full-text access In-app only via publisher TDM deals (not on API) Only where Semantic Scholar has OA full-text

What you get that PubMed keyword doesn't:

  1. Semantic recall. Keyword "F18 fimbriae" misses papers that say "K88 pilus" for the same protein; embeddings find them. This is the whole product — and it is why the semantic sources hit z=13.1 corroboration and 17/29 gold recall vs 4–8 and 13/29 for the free keyword APIs. They pull the right papers to the top.
  2. Structured enrichment per hit — study type, citations, journal quartile, sometimes pre-extracted PICO fields. graffold can weight edges by evidence quality (a highly-cited RCT vs a lone case report).
  3. Claim-level signal (Consensus especially) — an aggregate verdict across returned papers is a second trust signal we don't yet exploit.

What they are NOT giving you:

  • No new papers. Same Semantic Scholar corpus as the free API.
  • No closed-source full-text. Publisher deals apply in-app, not via API — that's why their full-text rates on the API are 7–13% (see below), not higher.
  • No proprietary data. Everything derived from the same OA/abstract layer anyone else can see.

Why the full-text rate is 7–13%, not 100%

Source Full-text on API Ceiling
PubMed 15% Europe PMC OA subset
Europe PMC 10% Europe PMC OA subset (native)
Consensus 13% Semantic Scholar S2ORC (OA only)
Elicit 7% Their API returns abstracts + metadata; in-app full-text is not exposed

Biomedical literature is only ~30–40% open-access, so ~30% is the legal ceiling for any non-publisher-contracted API. Above that requires paid TDM contracts with Wiley / Elsevier / Springer per publisher — out of scope for a benchmark and unnecessary for the trust statistic (see below).

Abstracts appear to be enough for the trust statistic

A finding worth naming: the Sept 21 run hit z=13.1 corroboration and 17/29 gold recall on mostly abstracts (~11% full-text overall, ~1,800 chars avg). Abstracts are dense — the punchline sentences, mechanism claims, and effect directions live there. Full-text adds methods and discussion prose, yielding more entities but also more noise, at ~15× the tokens per paper.

For methods-level detail (exact dosing, cell line, assay type) full-text is required. For corroboration density — what powers the trust signal on this page — abstracts appear sufficient. That is what the benchmark actually shows.

The strategic implication

Elicit/Consensus's ranking layer is a well-tuned LLM wrapper over a corpus graffold could query directly. The ranking is real and measurable — the results above would not exist without it — but the moat is the KG that consumes the ranked hits, not the ranker itself. Pay for a semantic source today; a graffold-ingest connector for Semantic Scholar + embeddings would capture the same retrieval quality without the per-seat license.

Reproduce

# 1. cache retrieval per source (free; no LLM)
ELICIT_API_KEY=... CONSENSUS_API_KEY=... NCBI_API_KEY=... python benchmarks/source_eval.py \
  --queries queries.txt --sources pubmed,europepmc,elicit,consensus --limit 10 --out /tmp/eval
# 2. extract + null tests + weighted graph viz (Bedrock Haiku)
AWS_PROFILE=default AWS_REGION=us-east-1 python benchmarks/source_eval.py \
  --queries queries.txt --sources pubmed,europepmc,elicit,consensus --limit 10 \
  --extract --null --viz --service bedrock \
  --model global.anthropic.claude-haiku-4-5-20251001-v1:0 --out /tmp/eval
# 3. summary plots (radar / scatter / recurrence)
python benchmarks/summary_plots.py /tmp/eval/results.json /tmp/eval/summary --html

Elicit needs a Pro-plan ELICIT_API_KEY; Consensus a CONSENSUS_API_KEY. Free sources need no key. Cached docs_*.json let retrieval and extraction run as separate steps.

Scope

Single domain (animal-health target profiles), single run, Haiku extraction. Directional, not definitive — the relative ordering (paid semantic sources lead on corroboration-above-chance and gold recall; all sources beat chance) is the finding. Re-run on your own queries before a purchasing decision.

Cost and scaling

What the benchmark itself cost

~$80 in Bedrock Haiku 4.5 for the full 4-source, 80-query, 2,413-paper run (Sept 16–21). Retrieval was ~free (Europe PMC / PubMed unpriced; Elicit and Consensus quota-bound). LLM extraction is the whole cost. Per-paper rate: ~$0.033/paper at this run's mix (~11% full-text, ~1,800 chars avg).

All extrapolations below are Haiku-4.5-based

The per-paper rates are derived from the real Haiku 4.5 run. Frontier models are dramatically more expensive on the same corpus:

Model ~$/M in ~$/M out 100k today's mix 100k full-text
Haiku 4.5 (what we ran) $1 $5 ~$3,300 ~$50,000
Sonnet 4.6 $3 $15 ~$10,000 ~$150,000
Opus 4.5 $15 $75 ~$50,000 ~$750,000
GPT-5 (est.) $3 $12 ~$8,500 ~$130,000

Haiku is the production choice for structured extraction; Sonnet/Opus are reserved for hard reasoning steps, not the bulk-extraction path.

Extrapolating to 100k papers

Corpus mix Per-paper 100k total Wall time (conc=10)
Today's mix (~11% full-text) $0.033 ~$3,300 ~24 hr
All abstracts ~$0.025 ~$2,500 ~18 hr
50% full-text ~$0.15 ~$15,000 ~4 days
All full-text (~25k chars avg) ~$0.50 ~$50,000 ~2 weeks

Bedrock TPM limits will bind before wall-time does at 100k+; realistic throughput needs concurrency + quota bumps.

Self-hosted alternative: 2× RTX 3090 + Llama 3.3 70B

An open 70B (Llama 3.3 70B, Qwen 2.5 72B) at 4-bit AWQ fits on a 2× RTX 3090 rig with headroom for structured KG extraction. Rough numbers:

Line item Cost
2× used RTX 3090 (24 GB each) ~$1,400
Motherboard + CPU + 64 GB RAM + 1500 W PSU + case ~$1,100
Total capex ~$2,500
Amortized over 3 years $2.28/day
Power (500 W avg × 24 h × $0.12/kWh) $1.44/day
Running cost, all-in ~$110/mo

Throughput ~300–500 papers/hour on structured extraction, so 100k papers is ~10 days of continuous inference. Breakeven vs Bedrock: ~40k papers/year (the $1,320 fixed annual cost divided by Haiku's $0.033/paper). If you refresh a 100k-paper KG quarterly, you cross breakeven in month 4.

Quality gap: open 70Bs at 4-bit are ~10–15% behind Haiku 4.5 on entity / relationship recall in structured extraction — measurable but not disqualifying. For the trust-statistic pipeline that gap is partially absorbed by corroboration (a claim needs ≥2 sources, so isolated recall misses don't survive to the KG anyway).

Rented GPU: the sweet spot for most cases

On-demand 2×-3090-class instances (RunPod, Vast.ai) run ~$0.50–0.90/hr. At 400 papers/hr that's ~$0.002–0.003/paper — roughly the same as owned after amortization, but with no capex, no rig-death risk, no idle burn. For occasional benchmarks or when 70B quality moves, this is usually the right default.

Summary

Path Setup 100k papers Wall time Best for
Bedrock Haiku 4.5 $0 ~$3,300 ~1 day latency, occasional runs, top quality
Owned 2× 3090 + 70B ~$2,500 ~$200 electricity ~10 days continuous ingestion; the GPUs do other work too
Rented 2× 3090 + 70B $0 ~$175 ~10 days scheduled batch runs; no capex

Bedrock still wins on latency and top-tier quality; the self-hosted paths win on $/paper past ~40k papers/year of steady throughput.

Assumptions worth checking

Rig numbers assume 3-year amortization, $0.12/kWh, 24/7 utilization, and Llama 3.3 70B AWQ throughput on 2× 3090 measured on comparable structured extraction workloads. Quality gap is a rough industry number for open 70Bs vs Haiku 4.5 on structured JSON extraction, not measured on our pipeline. Numbers are directional — run one small self-hosted arm before committing capex.