Evidence Weighting

Comprehensive guide to evidence quality scoring and weighting in the knowledge graph for biologist agent decision support.

Overview

Evidence weighting assigns quality scores to protein-disease associations based on study design, sample size, source type, and replication. This enables the biologist agent to prioritize high-quality evidence when answering queries.

Weighting Factors

Evidence quality is calculated from four components:

1. Study Design - Research methodology quality (0.3-1.0)
2. Source Type - Data source reliability (0.3-1.0)
3. Sample Size - Statistical power adjustment (±0.1)
4. Retraction Status - Quality control penalty (0.0-1.0)

Combined Weight Formula

final_weight = (
    study_design_weight 
    × source_type_weight 
    × sample_size_adjustment 
    × retraction_penalty
)

Study Design Weighting

Hierarchy

Study Design	Weight	Description
Randomized Controlled Trial (RCT)	1.0	Gold standard - random assignment to treatment/control
Meta-Analysis	0.95	Statistical pooling of multiple studies
Systematic Review	0.90	Comprehensive review of existing studies
Cohort Study	0.70	Prospective follow-up over time
Case-Control Study	0.60	Retrospective comparison of cases vs controls
Observational Study	0.50	Observation without intervention
Unknown	0.40	Cannot determine from abstract
Case Report	0.30	Individual patient case description

Classification

Study design is classified using LLM-based analysis:

from src.processors.evidence_scorer import EvidenceScorer
from src.factories.llm_factory import get_llm

llm = get_llm(service="local")
scorer = EvidenceScorer(llm=llm)

# Classify study design
result = await scorer.score_study_design(abstract_text)

print(f"Study Design: {result.study_design.study_type}")
print(f"Weight: {result.study_design.weight}")
print(f"Confidence: {result.confidence}")
print(f"Reasoning: {result.reasoning}")

Validation

Target accuracy: ≥80% on validation samples

# Run validation
python tests/performance/validate_evidence_scorer.py

Expected output:

VALIDATION SUMMARY
Total samples:  5
Correct:        5
Accuracy:       100.0%
✓ PASS: Classification accuracy ≥80%

Source Type Weighting

Hierarchy

Source Type	Weight	Rationale
Curated Databases	1.0	Expert-reviewed, high quality
Full-Text Articles	0.9	Complete methodology and results
PubMed Abstracts	0.8	Peer-reviewed but limited detail
Preprints	0.6	Not peer-reviewed yet
Social Media	0.3	Unverified, low reliability

Source Attribution

// Store source type on relationships
MATCH (p:Protein)-[r:ASSOCIATES_WITH]->(d:Disease)
SET r.source_type = 'full_text',  // or 'abstract', 'preprint', 'curated'
    r.source_type_weight = 0.9

Sample Size Adjustment

Adjustment Rules

Sample Size	Adjustment	Rationale
n ≥ 1000	+0.1	Large studies, high statistical power
100 ≤ n < 1000	0.0	Medium studies, adequate power
50 ≤ n < 100	-0.05	Small studies, limited power
n < 50	-0.1	Very small studies, low power
Unknown	0.0	No adjustment

Extraction

Sample size is extracted using LLM-based analysis:

# Extract sample size
extraction = await scorer.extract_sample_size(abstract_text)

print(f"Sample Size: {extraction.sample_size}")  # e.g., 1200
print(f"Confidence: {extraction.confidence}")
print(f"Reasoning: {extraction.reasoning}")

Validation

# Run sample size extraction validation
python tests/performance/validate_sample_size_extraction.py

Retraction Penalty

Penalty Levels

Retraction Nature	Penalty	Final Weight
Full Retraction	0.0	0.0 (excluded)
Expression of Concern	0.3	weight × 0.3
Correction	0.8	weight × 0.8
Not Retracted	1.0	No change

Retraction Checking

from src.processors.retraction_watch_checker import RetractionWatchChecker

checker = RetractionWatchChecker(
    db=db,
    backend="neo4j",  # or "memgraph"
    api_key="your-api-key"
)

# Check for retractions
await checker.check_retractions(pmids=["12345678"])

Graph Flags

// Flag retracted papers
MATCH (a:Abstract {pmid: "12345678"})
SET a.is_retracted = true,
    a.retraction_nature = "Retraction",
    a.retraction_date = "2024-01-15",
    a.retraction_reason = "Data fabrication"

// Flag affected relationships
MATCH (p:Protein)-[r:ASSOCIATES_WITH]->(d:Disease)
WHERE "12345678" IN r.pmids
SET r.is_from_retracted_paper = true

Confidence Score Aggregation

Single Study Confidence

confidence = scorer.calculate_evidence_confidence(
    study_design=StudyDesign.RCT,  # Base: 1.0
    sample_size=500,  # +0.1 (n≥100)
    replication_count=1  # No bonus
)
# Result: 1.0 (capped at 1.0)

Multi-Study Confidence

confidence = scorer.calculate_evidence_confidence(
    study_design=StudyDesign.COHORT,  # Base: 0.7
    sample_size=150,  # +0.1 (n≥100)
    replication_count=3  # +0.10 (2 additional studies)
)
# Result: 0.9 (0.7 + 0.1 + 0.1)

Formula

confidence = (
    study_design_weight  # Base score (0.3-1.0)
    + sample_size_bonus  # +0.1 for n≥100, +0.05 for n≥50
    + replication_bonus  # +0.05 per additional study (max +0.15)
)
# Capped at 1.0

Example Calculations

Study Design	Sample Size	Replication	Base	Sample Bonus	Replication Bonus	Final
RCT	200	1	1.0	+0.1	0	1.0 (capped)
Cohort	75	2	0.7	+0.05	+0.05	0.8
Case-Control	30	3	0.6	0	+0.10	0.7
Meta-Analysis	500	5	0.95	+0.1	+0.15	1.0 (capped)

Relationship Consolidation

Aggregate Evidence Confidence

During relationship consolidation, evidence confidence is aggregated across multiple sources:

from src.processors.relationship_counter import RelationshipCounter

counter = RelationshipCounter(db=db, backend="neo4j")
stats = counter.consolidate_comprehensive_relationships(dry_run=False)

# Consolidated relationships have:
# - evidence_confidence: Aggregate confidence (0.0-1.0)
# - avg_sample_size: Average sample size across studies
# - study_count: Number of studies with quality data

Weighted Average

def _calculate_aggregate_evidence_confidence(
    study_designs: List[str],
    sample_sizes: List[int],
    source_types: List[str]
) -> float:
    """
    Aggregate confidence using weighted average.

    confidence = Σ(study_weight × source_weight × sample_adjustment) / n
    """
    total_confidence = 0.0
    for design, size, source in zip(study_designs, sample_sizes, source_types):
        study_weight = get_study_design_weight(design)
        source_weight = get_source_type_weight(source)
        sample_adjustment = get_sample_size_adjustment(size)

        confidence = study_weight * source_weight * sample_adjustment
        total_confidence += confidence

    return min(total_confidence / len(study_designs), 1.0)

Graph Storage

Relationship Properties

// Store evidence quality on relationships
MATCH (p:Protein)-[r:COMPREHENSIVE_INTERACTION]->(d:Disease)
SET r.study_design = 'randomized_controlled_trial',
    r.study_design_weight = 1.0,
    r.sample_size = 1200,
    r.sample_size_adjustment = 0.1,
    r.source_type = 'full_text',
    r.source_type_weight = 0.9,
    r.is_from_retracted_paper = false,
    r.retraction_penalty = 1.0,
    r.evidence_confidence = 0.99,  // Aggregate confidence
    r.final_weight = 0.99  // Combined weight

Query by Quality

// Find high-quality evidence (RCT or meta-analysis)
MATCH (p:Protein)-[r:COMPREHENSIVE_INTERACTION]->(d:Disease)
WHERE r.study_design_weight >= 0.9
  AND r.is_from_retracted_paper = false
RETURN p.name, d.name, r.study_design, r.evidence_confidence
ORDER BY r.evidence_confidence DESC

// Find large studies (n≥1000)
MATCH (p:Protein)-[r:COMPREHENSIVE_INTERACTION]->(d:Disease)
WHERE r.sample_size >= 1000
RETURN p.name, d.name, r.sample_size, r.study_design
ORDER BY r.sample_size DESC

// Exclude retracted evidence
MATCH (p:Protein)-[r:COMPREHENSIVE_INTERACTION]->(d:Disease)
WHERE r.is_from_retracted_paper = false
RETURN p.name, d.name, r.evidence_confidence

Biologist Agent Integration

Query Synthesis

The biologist agent uses evidence weights to prioritize high-quality evidence:

from src.agents.biologist_agent import BiologistAgent

agent = BiologistAgent(db=db, llm=llm)

# Query with evidence weighting
response = await agent.query(
    question="What proteins are associated with Alzheimer's disease?",
    min_evidence_confidence=0.7,  # Only high-quality evidence
    exclude_retracted=True  # Exclude retracted papers
)

print(response.answer)
print(f"Sources: {len(response.sources)}")
print(f"Average confidence: {response.avg_confidence}")

Evidence Summary

# Get evidence quality summary
summary = agent.get_evidence_summary(
    protein="APOE",
    disease="Alzheimer disease"
)

print(f"Total studies: {summary.total_studies}")
print(f"RCTs: {summary.rct_count}")
print(f"Meta-analyses: {summary.meta_analysis_count}")
print(f"Average sample size: {summary.avg_sample_size}")
print(f"Retracted papers: {summary.retracted_count}")
print(f"Overall confidence: {summary.overall_confidence}")

Testing

Unit Tests

# Run evidence scorer tests
uv run pytest tests/unit/test_evidence_scorer.py -v

Coverage: - ✅ Study design classification (all types) - ✅ Sample size extraction (found, not found, large/small) - ✅ Confidence score calculation (single and multi-study) - ✅ Weight retrieval and mapping - ✅ Error handling (invalid JSON, LLM errors)

Integration Tests

# Run study design validation
python tests/performance/validate_evidence_scorer.py

# Run sample size extraction validation
python tests/performance/validate_sample_size_extraction.py

Example Scenarios

Scenario 1: High-Quality Evidence

Study: RCT with 1,500 participants
Source: Full-text article
Retraction: Not retracted

Calculation:
- Study design weight: 1.0 (RCT)
- Source type weight: 0.9 (full-text)
- Sample size adjustment: +0.1 (n≥1000)
- Retraction penalty: 1.0 (not retracted)

Final weight = 1.0 × 0.9 × 1.1 × 1.0 = 0.99
Evidence confidence = 1.0 (capped)

Scenario 2: Medium-Quality Evidence

Study: Cohort study with 250 participants
Source: PubMed abstract
Retraction: Not retracted

Calculation:
- Study design weight: 0.7 (cohort)
- Source type weight: 0.8 (abstract)
- Sample size adjustment: 0.0 (100≤n<1000)
- Retraction penalty: 1.0 (not retracted)

Final weight = 0.7 × 0.8 × 1.0 × 1.0 = 0.56
Evidence confidence = 0.7 (base)

Scenario 3: Low-Quality Evidence

Study: Case report with 5 participants
Source: Preprint
Retraction: Not retracted

Calculation:
- Study design weight: 0.3 (case report)
- Source type weight: 0.6 (preprint)
- Sample size adjustment: -0.1 (n<50)
- Retraction penalty: 1.0 (not retracted)

Final weight = 0.3 × 0.6 × 0.9 × 1.0 = 0.162
Evidence confidence = 0.2 (0.3 - 0.1)

Scenario 4: Retracted Evidence

Study: RCT with 1,000 participants
Source: Full-text article
Retraction: Full retraction

Calculation:
- Study design weight: 1.0 (RCT)
- Source type weight: 0.9 (full-text)
- Sample size adjustment: +0.1 (n≥1000)
- Retraction penalty: 0.0 (retracted)

Final weight = 1.0 × 0.9 × 1.1 × 0.0 = 0.0
Evidence confidence = 0.0 (excluded)

Troubleshooting

Low Classification Accuracy

Problem: Study design classification <80% accurate

Solutions: 1. Use larger LLM models for better accuracy 2. Add more examples to validation set 3. Fine-tune prompt based on failure patterns 4. Verify abstracts contain methodology information

Sample Size Extraction Errors

Problem: Incorrect or missing sample sizes

Solutions: 1. Check if abstract contains sample size information 2. Verify LLM distinguishes sample size from other numbers 3. For meta-analyses, ensure total participants extracted 4. Add more examples to prompt for edge cases

Confidence Score Issues

Problem: Confidence scores don't match expectations

Solutions: 1. Verify study design weights are correct 2. Check sample size adjustment logic 3. Ensure replication bonus is calculated correctly 4. Verify confidence is capped at 1.0

Related Documentation

• EVIDENCE_SCORING.md - Detailed evidence scoring implementation
• RELATIONSHIP_COUNTER.md - Relationship consolidation with evidence
• RETRACTION_WATCH_INTEGRATION.md - Retraction checking
• DATABASE_INTEGRATIONS.md - Curated database integrations

Future Enhancements

• P-Value Extraction: Extract statistical significance from results
• Effect Size Extraction: Extract odds ratios, hazard ratios, etc.
• Publication Bias Detection: Identify potential publication bias
• Temporal Weighting: Newer studies weighted higher
• Journal Impact Factor: Weight by journal quality
• Author Reputation: Weight by author h-index