KG Value Study — reproduce-it evidence (reliability, speed, cost)¶
This is the technical, reproduce-it companion to the Pitch's "Measured value" section. Same findings, with the raw per-run numbers, the disproven theses in full, and the scripts to re-run it yourself.
Real-run evidence for how the graffold knowledge graph adds value to a discovery
pipeline. Numbers are from live runs (Bedrock DeepSeek-v3.2, keyed/keyless
PubMed, KGs served from ~/.graffold/parquet/<kg_id>/ via the /v1/kg API),
from the biot integration study (2026-09).
TL;DR scorecard¶
| Axis | Result | Status |
|---|---|---|
| Reliability | model flips validated/killed verdict ~25% of runs; KG 0% | DEMONSTRATED |
| Speed | KG grounding 1.8x faster than live PubMed research | DEMONSTRATED |
| Cost | KG 5.7x fewer tokens/query, 5.6 PubMed calls avoided | DEMONSTRATED |
| Novel insight | a strong model out-generates the KG's cross-disease bridges | DISPROVEN |
| Correctness | corroboration proven non-random (agreement null, p≤0.001) and predictive (held-out recurrence 8–14×); ground-truth still unlabelled | STRONG PROXY |
| Provenance | evidence_ref per edge shipped; ≥2-source support now measured per corpus (5.1–6.1% on ETEC — see Source Comparison Study) |
MEASURED |
The defensible pitch today is reliability + speed + cost + corroboration-as-proxy — NOT "novel insight." The KG is reproducible and cheaper now; corroboration is proven non-random and predictive of recurrence (a strong label-free proxy for correctness), though ground-truth labelling is still owed.
Study 1 — Reliability (verdict stability)¶
Same question asked N=7 times to each side: "Is target X validated or killed for disease D?"
| disease | MODEL flip rate | KG flip rate |
|---|---|---|
| zoetis-mastitis | 29% | 0% |
| elanco-coccidiosis | 29% | 0% |
| alltech-mucin | 21% | 0% |
The model (temp 0.7) returns a different verdict on ~1 in 4 targets across
identical prompts. The KG's validate_targets verdict is a deterministic query
— bit-identical every run. For an auditable pipeline, non-deterministic
kill/advance calls are a liability; the KG makes the verdict layer stable.
Study 2 — Speed / cost (n=30 per arm, 3 diseases)¶
Same target-proposal question, three arms:
| arm | latency mean | tokens/query | PubMed calls |
|---|---|---|---|
| kg (KG-grounded) | 18.4s | 6,928 | 0 |
| research (live PubMed) | 33.3s | 39,179 | 5.6 |
| control (neither) | 17.0s | 4,058 | 0 |
KG vs research: ~1.8x faster, ~5.7x fewer tokens (32k saved/query), ~6 PubMed round-trips avoided. Research is a fixed cost paid ONCE at ingestion; the KG amortizes it to a cheap reusable lookup. KG wins on cost from the first reuse.
From "owed" to "strong proxy" — corroboration is non-random and predictive¶
The original correctness IOU was: an edge asserted by many independent sources should be more trustworthy than one asserted once. That is now measured and tested against chance, not just asserted. Full per-source numbers (and the weighted graph viz) are in the Source Comparison Study; the evidence chain:
1. ≥2-source support is real and varies by corpus. Building KGs from the
same queries across literature sources, the fraction of edges backed by claims
recurring across distinct papers is non-trivial and source-dependent
(5.1–6.1% on the ETEC corpus) — a real signal, not a constant.
2. Agreement null — corroboration beats chance (p≤0.001). Under a null where each paper asserts the same number of edges drawn at random from its own vocabulary (holding entity popularity + per-paper edge count fixed), real ≥2-source support exceeds the null by z = 5–11, p = 0.001 for every source. Different papers independently assert the same specific relationship far more than shared vocabularies would produce by chance.
Honest process note: this took three null models. The first two drew each witness from the whole paper pool, inflating the null (a 5-witness edge almost always hits ≥2 papers by chance) and producing a false negative. A diagnostic showed the real confound — multi-source edges sit between ~2× higher-degree entities — so the correct null had to hold each paper's vocabulary fixed. The null must control the confound the data actually has.
3. Held-out recurrence — support PREDICTS truth (8–14× lift). The closest label-free proxy for truth: split papers into halves A/B and ask whether edges corroborated in A recur in the independent half B. Corroborated edges recur 54–61% of the time; single-source edges only 4–6% — an 8–14× lift (and the lift strengthens at 10× multi-topic scale). Support doesn't just count witnesses; it predicts which relationships are real enough to recur in evidence the extractor never saw.
4. Contradictions are a FEATURE. Contradicted edges carry ~3–4× the mean
support of ordinary edges — because genuine scientific disputes are the
well-studied, heavily-witnessed relationships. Classified by whether opposing
claims cite different papers (genuine dispute) or the same paper (extraction
noise), the large majority are genuine disputes with provenance on each side.
The KG surfaces controversy with citations (via /v1/kg/contradictions)
rather than averaging it away — exactly the risk intelligence a discovery
pipeline needs.
What this establishes and what it doesn't. Corroboration is proven non-random and predictive of recurrence — a strong, defensible, correctness-adjacent claim. It does NOT yet assert ground-truth membership; that needs a labelled gold set. The honest sequence is now: reproducible + cheap (demonstrated) → traceable (provenance shipped) → corroboration non-random + predictive (demonstrated) → ground-truth correctness (still owed).
The measured numbers (incl. Consensus)¶
The corroboration mechanism was first measured on a small-sample run (creatine, ~40 papers/source) that did include Consensus before its free-tier quota lapsed. Consensus posted the strongest KG-refinement of the three:
KG structure — density and corroboration per source:
| Source | Entities | Density | ≥2-source support |
|---|---|---|---|
| pubmed | 489 | 1.46 | 5.1% |
| europepmc | 281 | 1.51 | 8.3% |
| consensus | 285 | 1.63 | 10.8% |
Agreement null — real ≥2-source support vs a vocabulary-preserving null:
| Source | real ≥2-src | null mean | z | p |
|---|---|---|---|---|
| pubmed | 48 | 14.0 | +10.6 | 0.0005 |
| europepmc | 38 | 26.3 | +3.4 | 0.001 |
| consensus | 61 | 27.5 | +8.8 | 0.0005 |
Held-out recurrence — does support predict recurrence in the other half:
| Source | ≥2-src recurs | single-src recurs | lift |
|---|---|---|---|
| pubmed | 54.5% | 5.9% | 9.2× |
| europepmc | 73.8% | 6.9% | 10.6× |
| consensus | 60.3% | 11.4% | 5.3× |
Consensus builds the leanest, most-corroborated graph and beats chance strongly (z=8.8) — the same signature the larger ETEC run showed for Elicit (the other paid semantic source). Consensus's held-out lift is lowest only because its single-source baseline is already the highest (11.4%) — even its weak edges recur more. The larger multi-topic re-run (Source Comparison Study) deferred Consensus on the 30-search/month free cap; a Pro key restores it at scale.
What was DISPROVEN (report honestly)¶
The "KG surfaces novel cross-disease insight a model can't" thesis failed on well-published diseases: a control model with no KG spontaneously generated 12 defensible non-obvious cross-disease target bridges (PDE10A, STING, LOXL2, etc.). The KG's specific bridges (SLC13A3, galectin-3) were real but the model produced equally-good ones unaided. On heavily-modeled diseases the KG does not beat the model's generation. Value must come from reliability/cost/provenance, or from diseases/decisions the model CANNOT know (proprietary negatives).
What is NOT yet claimed — ground-truth correctness¶
Corroboration being non-random and predictive (above) is a strong proxy, not a ground-truth guarantee. Reliability measures consistency, not truth: the KG returns the same verdict every run, but whether that verdict is RIGHT still depends on ingestion quality — a stale or thinly-sourced KG will reliably return a wrong answer. Asserting ground-truth correctness requires a labelled gold edge set showing high-support edges are true more often than low-support ones; the held-out recurrence test is the closest label-free stand-in until then.
Honest caveats (do not omit from the wiki)¶
- Reliability measures CONSISTENCY, not CORRECTNESS — the KG is reliably consistent; whether its verdicts are RIGHT depends on ingestion quality.
- Cost is per-query, not quality — cheaper only wins if the answer is good enough.
- The kg arm still spends ~7k tokens (model reasoning over KG context). The pure
validate_targetsDB call is ~0.3ms/~0 tokens but is verdict-only. - Model flip rate is temperature-dependent (0.7 here).
Reproduce¶
Portable scripts in benchmarks/ (need only boto3 + a served KG API, no atlas):
- kg_reliability.py <disease> <kg_port> <n> — verdict flip-rate, model vs KG.
- score_speedcost.py / score_night.py — aggregate metric scorers.
Serve a KG: ATLAS_DEFAULT_KG=<kg_id> uvicorn biot_api.main:app --port 8050
(from the biot repo). The speed/cost runner itself is atlas-coupled (uses the
atlas native agent loop for the research arm) and lives in the atlas repo.