Which literature source builds the best knowledge graph?¶
For building a knowledge graph with graffold, does a paid semantic-search source (Elicit, Consensus) produce a better, more trustworthy KG than the free PubMed / Europe PMC APIs — and if so, which one is worth paying for?
We answer with one fair experiment: identical queries, identical pipeline, only the source changes. Canonical run: 80 queries across four animal-health target profiles, ~550–700 papers/source, ~3,500–4,600 entities each — now including a full-scale Consensus arm.
Bottom line
The two paid semantic sources (Elicit, Consensus) both win — and they win differently. Both retrieve an almost entirely different slice of the literature (so they stack with the free APIs), and both post the strongest corroboration-above-chance (Consensus z=13.1, Elicit z=12.8 — vs 4–8 for the free sources). Elicit builds the leanest graph (fewest entities, highest ≥2-source rate at 6.8%); Consensus retrieves the most and recalls the most gold (700 papers, 17/29 named targets). Keep the free sources for breadth; add a paid semantic source for the corroboration-dense core.
The story in three plots¶
Elicit's polygon is largest — it leads on corroboration density and uniqueness and is the leanest. Consensus is close behind with the strongest above-chance signal and widest reach. The free sources are solid but smaller on every axis.
The best KG sits upper-left: fewer entities (less noise) and higher ≥2-source support (more corroboration). Elicit is there — ~15% fewer entities than PubMed with the highest corroboration.
The key result: split the papers in half; a corroborated edge (≥2 papers) recurs in the independent other half ~60% of the time, a single-source edge only 5–7%. Support isn't just a count — it predicts which relationships are real. Holds for every source.
The numbers¶
80 queries, 4 animal-health target profiles, Bedrock Claude Haiku extraction. All four sources run at full scale.
What is ≥2-source support?
The fraction of edges (relationships) in the KG that are asserted by claims from at least 2 distinct source papers.
Concretely, for each edge (e.g. creatine — increases → muscle_mass),
count how many distinct PMIDs (or paper IDs) contain a claim producing
that edge. Then:
Why it matters: it's the KG's trust statistic. A single paper claiming X causes Y is a lead; two independent papers claiming the same thing is a corroborated claim. The higher this rate, the more of your graph is backed by more than one witness.
In this study: Elicit's KG had 6.8% ≥2-source edges — highest of the four. That means for every 100 edges Elicit's KG builds, ~7 are corroborated by ≥2 papers vs ~5 for Europe PMC. Combined with its lean size (fewest entities), Elicit gives the highest density of corroborated signal per entity.
The null test (next column) then checks: is 6.8% actually above chance, or would you get that just from popular entities co-occurring? Answer for Elicit (z=12.8, p=0.001): well above chance — the corroboration is real, not an artifact of degree.
| Source | Papers | Entities | ≥2-source support | Corroboration above chance | Held-out recurrence | Gold recall |
|---|---|---|---|---|---|---|
| PubMed | 546 | 3977 | 6.1% | z=7.7 (p=0.001) | 58% vs 5% single | 13/29 |
| Europe PMC | 574 | 4120 | 5.0% | z=4.1 (p=0.001) | 61% vs 4% single | 13/29 |
| Elicit | 593 | 3569 | 6.8% | z=12.8 (p=0.001) | 59% vs 7% single | 16/29 |
| Consensus | 700 | 4637 | 6.3% | z=13.1 (p=0.001) | 57% vs 6% single | 17/29 |
- ≥2-source support = fraction of edges backed by a claim in ≥2 distinct papers. This is the KG's trust statistic. Elicit leads (6.8%) — leanest graph, densest corroboration per entity.
- Above chance = an agreement-null test (each paper asserts random edges among its own vocabulary); real corroboration beats it at p=0.001 everywhere, strongest for the two paid semantic sources (Consensus z=13.1, Elicit z=12.8) vs the free APIs (z=4–8).
- Held-out = does a corroborated edge reappear in an independent half of the literature? ~57–61% do; single-source edges only 4–7%.
- Gold recall = of 29 named target-profile entities, how many the source recovered from literature alone, unseeded. Consensus 17, Elicit 16 — the semantic sources beat the free APIs (both 13).
- Retrieval overlap (title-normalized): PubMed∩Europe PMC 15%; the semantic sources are 1–7% with the free APIs — genuinely different, complementary slices. Elicit∩Consensus overlap only 13%, so even the two paid sources largely complement each other.
See the graphs¶
Each source's knowledge graph, weighted by corroboration — thick/green edges are backed by multiple papers, thin/grey are single-source. Flip the tabs; Elicit visibly has more green per node.
Which to pay for, and why it helps graffold¶
- Both paid semantic sources beat the free APIs on the signal that matters — corroboration above chance (Consensus z=13.1, Elicit z=12.8 vs z=4–8) and gold recall (17, 16 vs 13). Their semantic ranking pulls the on-topic, well-witnessed papers that make the trust statistic meaningful.
- Elicit for the leanest, densest graph; Consensus for reach. Elicit builds the fewest entities with the highest ≥2-source rate (6.8%) — less noise per corroborated edge. Consensus retrieves the most papers (700) and recalls the most gold (17/29). They overlap only ~13% with each other, so running both is additive, not redundant.
- Keep the free sources. The semantic sources overlap PubMed/Europe PMC only ~1–7%, so the strongest KG uses all of them — free for breadth, paid for the corroboration-dense core. Corroboration counts across distinct sources, so more independent sources directly raise the trust signal.
- Enrichment is a bonus graffold can use — Elicit/Consensus attach study-type, citation count, and journal quartile per paper, letting graffold weight edges by evidence quality (a heavily-cited RCT vs a lone case report).
Contradictions are a feature, not noise: contradicted edges carry ~3× the
support of ordinary ones because genuine scientific disputes are the
heavily-studied relationships — graffold surfaces them with per-side provenance
via /v1/kg/contradictions
rather than averaging them away.
What Elicit and Consensus are actually selling¶
A fair question after seeing the numbers: what are you paying for? Both Elicit and Consensus are LLM curation layers on top of the same corpus everyone else has (Semantic Scholar, ~200M papers). They don't sell access — they sell semantic ranking, per-paper enrichment, and claim-level summarization.
| Elicit | Consensus | |
|---|---|---|
| Underlying corpus | Semantic Scholar (~200M) | Semantic Scholar (~200M) |
| Semantic ranking | Embeddings + query rewriting, tuned for research questions | Same, tuned for scientific claims |
| Per-paper LLM extraction | Population / intervention / outcome / n / effect direction ("columns") | Yes / No / Nuanced verdict on the query claim |
| Metadata enrichment | Study type, citation count, journal quartile | Same + "consensus meter" aggregation |
| Content returned to API | Abstracts + extracted fields | Abstracts + verdict + snippet |
| Full-text access | In-app only via publisher TDM deals (not on API) | Only where Semantic Scholar has OA full-text |
What you get that PubMed keyword doesn't:
- Semantic recall. Keyword
"F18 fimbriae"misses papers that say"K88 pilus"for the same protein; embeddings find them. This is the whole product — and it is why the semantic sources hit z=13.1 corroboration and 17/29 gold recall vs 4–8 and 13/29 for the free keyword APIs. They pull the right papers to the top. - Structured enrichment per hit — study type, citations, journal quartile, sometimes pre-extracted PICO fields. graffold can weight edges by evidence quality (a highly-cited RCT vs a lone case report).
- Claim-level signal (Consensus especially) — an aggregate verdict across returned papers is a second trust signal we don't yet exploit.
What they are NOT giving you:
- No new papers. Same Semantic Scholar corpus as the free API.
- No closed-source full-text. Publisher deals apply in-app, not via API — that's why their full-text rates on the API are 7–13% (see below), not higher.
- No proprietary data. Everything derived from the same OA/abstract layer anyone else can see.
Why the full-text rate is 7–13%, not 100%¶
| Source | Full-text on API | Ceiling |
|---|---|---|
| PubMed | 15% | Europe PMC OA subset |
| Europe PMC | 10% | Europe PMC OA subset (native) |
| Consensus | 13% | Semantic Scholar S2ORC (OA only) |
| Elicit | 7% | Their API returns abstracts + metadata; in-app full-text is not exposed |
Biomedical literature is only ~30–40% open-access, so ~30% is the legal ceiling for any non-publisher-contracted API. Above that requires paid TDM contracts with Wiley / Elsevier / Springer per publisher — out of scope for a benchmark and unnecessary for the trust statistic (see below).
Abstracts appear to be enough for the trust statistic¶
A finding worth naming: the Sept 21 run hit z=13.1 corroboration and 17/29 gold recall on mostly abstracts (~11% full-text overall, ~1,800 chars avg). Abstracts are dense — the punchline sentences, mechanism claims, and effect directions live there. Full-text adds methods and discussion prose, yielding more entities but also more noise, at ~15× the tokens per paper.
For methods-level detail (exact dosing, cell line, assay type) full-text is required. For corroboration density — what powers the trust signal on this page — abstracts appear sufficient. That is what the benchmark actually shows.
The strategic implication¶
Elicit/Consensus's ranking layer is a well-tuned LLM wrapper over a corpus graffold could query directly. The ranking is real and measurable — the results above would not exist without it — but the moat is the KG that consumes the ranked hits, not the ranker itself. Pay for a semantic source today; a graffold-ingest connector for Semantic Scholar + embeddings would capture the same retrieval quality without the per-seat license.
Reproduce¶
# 1. cache retrieval per source (free; no LLM)
ELICIT_API_KEY=... CONSENSUS_API_KEY=... NCBI_API_KEY=... python benchmarks/source_eval.py \
--queries queries.txt --sources pubmed,europepmc,elicit,consensus --limit 10 --out /tmp/eval
# 2. extract + null tests + weighted graph viz (Bedrock Haiku)
AWS_PROFILE=default AWS_REGION=us-east-1 python benchmarks/source_eval.py \
--queries queries.txt --sources pubmed,europepmc,elicit,consensus --limit 10 \
--extract --null --viz --service bedrock \
--model global.anthropic.claude-haiku-4-5-20251001-v1:0 --out /tmp/eval
# 3. summary plots (radar / scatter / recurrence)
python benchmarks/summary_plots.py /tmp/eval/results.json /tmp/eval/summary --html
Elicit needs a Pro-plan ELICIT_API_KEY; Consensus a CONSENSUS_API_KEY. Free
sources need no key. Cached docs_*.json let retrieval and extraction run as
separate steps.
Scope
Single domain (animal-health target profiles), single run, Haiku extraction. Directional, not definitive — the relative ordering (paid semantic sources lead on corroboration-above-chance and gold recall; all sources beat chance) is the finding. Re-run on your own queries before a purchasing decision.
Cost and scaling¶
What the benchmark itself cost¶
~$80 in Bedrock Haiku 4.5 for the full 4-source, 80-query, 2,413-paper run (Sept 16–21). Retrieval was ~free (Europe PMC / PubMed unpriced; Elicit and Consensus quota-bound). LLM extraction is the whole cost. Per-paper rate: ~$0.033/paper at this run's mix (~11% full-text, ~1,800 chars avg).
All extrapolations below are Haiku-4.5-based
The per-paper rates are derived from the real Haiku 4.5 run. Frontier models are dramatically more expensive on the same corpus:
| Model | ~$/M in | ~$/M out | 100k today's mix | 100k full-text |
|---|---|---|---|---|
| Haiku 4.5 (what we ran) | $1 | $5 | ~$3,300 | ~$50,000 |
| Sonnet 4.6 | $3 | $15 | ~$10,000 | ~$150,000 |
| Opus 4.5 | $15 | $75 | ~$50,000 | ~$750,000 |
| GPT-5 (est.) | $3 | $12 | ~$8,500 | ~$130,000 |
Haiku is the production choice for structured extraction; Sonnet/Opus are reserved for hard reasoning steps, not the bulk-extraction path.
Extrapolating to 100k papers¶
| Corpus mix | Per-paper | 100k total | Wall time (conc=10) |
|---|---|---|---|
| Today's mix (~11% full-text) | $0.033 | ~$3,300 | ~24 hr |
| All abstracts | ~$0.025 | ~$2,500 | ~18 hr |
| 50% full-text | ~$0.15 | ~$15,000 | ~4 days |
| All full-text (~25k chars avg) | ~$0.50 | ~$50,000 | ~2 weeks |
Bedrock TPM limits will bind before wall-time does at 100k+; realistic throughput needs concurrency + quota bumps.
Self-hosted alternative: 2× RTX 3090 + Llama 3.3 70B¶
An open 70B (Llama 3.3 70B, Qwen 2.5 72B) at 4-bit AWQ fits on a 2× RTX 3090 rig with headroom for structured KG extraction. Rough numbers:
| Line item | Cost |
|---|---|
| 2× used RTX 3090 (24 GB each) | ~$1,400 |
| Motherboard + CPU + 64 GB RAM + 1500 W PSU + case | ~$1,100 |
| Total capex | ~$2,500 |
| Amortized over 3 years | $2.28/day |
| Power (500 W avg × 24 h × $0.12/kWh) | $1.44/day |
| Running cost, all-in | ~$110/mo |
Throughput ~300–500 papers/hour on structured extraction, so 100k papers is ~10 days of continuous inference. Breakeven vs Bedrock: ~40k papers/year (the $1,320 fixed annual cost divided by Haiku's $0.033/paper). If you refresh a 100k-paper KG quarterly, you cross breakeven in month 4.
Quality gap: open 70Bs at 4-bit are ~10–15% behind Haiku 4.5 on entity / relationship recall in structured extraction — measurable but not disqualifying. For the trust-statistic pipeline that gap is partially absorbed by corroboration (a claim needs ≥2 sources, so isolated recall misses don't survive to the KG anyway).
Rented GPU: the sweet spot for most cases¶
On-demand 2×-3090-class instances (RunPod, Vast.ai) run ~$0.50–0.90/hr. At 400 papers/hr that's ~$0.002–0.003/paper — roughly the same as owned after amortization, but with no capex, no rig-death risk, no idle burn. For occasional benchmarks or when 70B quality moves, this is usually the right default.
Summary¶
| Path | Setup | 100k papers | Wall time | Best for |
|---|---|---|---|---|
| Bedrock Haiku 4.5 | $0 | ~$3,300 | ~1 day | latency, occasional runs, top quality |
| Owned 2× 3090 + 70B | ~$2,500 | ~$200 electricity | ~10 days | continuous ingestion; the GPUs do other work too |
| Rented 2× 3090 + 70B | $0 | ~$175 | ~10 days | scheduled batch runs; no capex |
Bedrock still wins on latency and top-tier quality; the self-hosted paths win on $/paper past ~40k papers/year of steady throughput.
Assumptions worth checking
Rig numbers assume 3-year amortization, $0.12/kWh, 24/7 utilization, and Llama 3.3 70B AWQ throughput on 2× 3090 measured on comparable structured extraction workloads. Quality gap is a rough industry number for open 70Bs vs Haiku 4.5 on structured JSON extraction, not measured on our pipeline. Numbers are directional — run one small self-hosted arm before committing capex.