Incremental PubMed Update Guide¶

Overview¶

The incremental update system enables weekly knowledge graph maintenance by fetching and processing only new PubMed abstracts published in the last N days. This approach is optimized for speed (<15 minutes for ~100 abstracts) and prevents duplicate data through intelligent deduplication.

Key Features¶

Date-Filtered Queries: Fetch only abstracts from the last N days (default: 7)
PMID Deduplication: Automatically skips abstracts already in the database
Entity Consolidation: Merges new entities with existing nodes using UniProt/MONDO IDs
Relationship Consolidation: Prevents duplicate relationships while preserving all evidence
Incremental Embeddings: Generates embeddings only for new content
Performance Optimized: Completes in <15 minutes for 100 abstracts

Quick Start¶

Basic Usage¶

# Update with default settings (last 7 days, 100 abstracts max)
uv run python scripts/incremental_pubmed_update.py \
  --search-term "cardiovascular disease protein biomarker" \
  --database olink1

# Custom time window (last 14 days)
uv run python scripts/incremental_pubmed_update.py \
  --search-term "kidney disease biomarker" \
  --days 14 \
  --database olink1

# Use AWS Bedrock for LLM processing
uv run python scripts/incremental_pubmed_update.py \
  --search-term "cancer protein marker" \
  --service bedrock \
  --database olink1

Weekly Cron Job¶

# Add to crontab for weekly updates (every Monday at 2 AM)
0 2 * * 1 cd /path/to/graffold && uv run python scripts/incremental_pubmed_update.py --search-term "protein disease" --database olink1 >> logs/weekly_update.log 2>&1

How It Works¶

5-Step Pipeline¶

Fetch New Abstracts (2-3 min)
Builds date-filtered PubMed query: search_term AND ("YYYY/MM/DD"[Date - Publication] : "YYYY/MM/DD"[Date - Publication])
Queries PubMed API with date range
Filters out existing PMIDs using Neo4jDatabase.get_existing_pmids()
Returns only new abstracts not in database
Extract Entities & Relationships (5-8 min)
Processes abstracts through LLM-based extraction
Extracts proteins, diseases, and relationships
Uses token-based chunking with sentence boundary preservation
Stores entities and relationships in Neo4j
Entity Consolidation (1-2 min)
Merges new entities with existing nodes
Uses UniProt ID matching (fastest, highest accuracy)
Preserves all properties and provenance
Updates synonyms and canonical IDs
Relationship Consolidation (1-2 min)
Incremental consolidation: Only processes relationships from new abstracts
Identifies relationships by source_doc property (PMID)
Merges with existing relationships between same entities
Aggregates PMIDs, confidence scores, and evidence
Preserves all features (extraction methods, timestamps, sentence contexts)
Creates non-redundant relationship links with complete provenance
Generate Embeddings (2-3 min)
Generates embeddings only for new chunks
Generates embeddings only for nodes without embeddings
Uses existing embedding service configuration
Enables semantic search for new content

PMID Deduplication¶

The system uses the graph database's Abstract nodes to track processed PMIDs:

-- Cypher query works on both Neo4j and Memgraph
MATCH (a:Abstract)
WHERE exists((a)<-[:FROM_CHUNK]-(:Chunk)<-[:HAS_CHUNK]-(:Document)) 
   OR exists((a)-[:PROCESSED]->(:ProcessingMarker))
RETURN a.pmid AS pmid

Only PMIDs not in this set are fetched and processed.

Entity Consolidation Strategy¶

New entities are merged with existing nodes using:

UniProt ID Matching (proteins)
Exact match on uniprot_id property
Highest accuracy, fastest performance
Preserves gene symbols and synonyms
MONDO ID Matching (diseases)
Exact match on mondo_id property
Links to disease ontology hierarchy
Preserves disease classifications
Synonym Matching (fallback)
Case-insensitive synonym comparison
Handles abbreviations and aliases
Lower accuracy, used when IDs unavailable

Relationship Consolidation¶

The incremental update uses targeted relationship consolidation that only processes relationships from new abstracts. This is much faster than full consolidation and prevents duplicate edges.

How it works: 1. Identifies relationships by their source_doc property (PMID) 2. For each new relationship, checks if a relationship already exists between the same entities 3. If exists: Merges evidence (appends PMIDs, confidence scores, contexts) 4. If new: Creates consolidated relationship with initial evidence

Before (new abstract PMID 99999):

TP53 -[ASSOCIATES_WITH {
  source_docs: ["12345", "67890"],
  confidence_scores: [0.9, 0.85],
  evidence_sources: 2
}]-> Cancer

# New relationship from PMID 99999
TP53 -[ASSOCIATES_WITH {source_doc: "99999", confidence: 0.88}]-> Cancer

After incremental consolidation:

TP53 -[ASSOCIATES_WITH {
  source_docs: ["12345", "67890", "99999"],
  confidence_scores: [0.9, 0.85, 0.88],
  avg_confidence: 0.877,
  evidence_sources: 3,
  extraction_methods: ["llm", "llm", "llm"],
  sentence_contexts: [...],
  first_seen: "2025-01-15",
  last_seen: "2025-02-25"
}]-> Cancer

Key benefits: - Only processes relationships from new PMIDs (fast) - Preserves all evidence from all sources (complete provenance) - Prevents duplicate edges (clean graph) - Updates statistics (avg_confidence, evidence_sources)

Performance Benchmarks¶

Typical Performance (100 abstracts)¶

Step	Duration	Notes
Fetch abstracts	2-3 min	PubMed API latency
Extract entities	5-8 min	LLM processing (varies by service)
Consolidate entities	1-2 min	UniProt ID matching
Consolidate relationships	1-2 min	Cypher aggregation
Generate embeddings	2-3 min	Embedding model inference
Total	11-18 min	Target: <15 min

Performance Optimization Tips¶

Use Local Ollama for fastest LLM processing (no API latency)
Reduce max_results if time budget is tight (50 abstracts = ~8 min)
Use Bedrock/SageMaker for production (better quality, slightly slower)
Run during off-peak hours to avoid PubMed API throttling

Command-Line Options¶

python scripts/incremental_pubmed_update.py [OPTIONS]

Required:
  --search-term, -t TEXT    PubMed search query

Optional:
  --days, -d INT           Days to look back (default: 7)
  --max-results, -n INT    Max abstracts to process (default: 100)
  --service, -s TEXT       LLM service: local|openai|bedrock|sagemaker (default: local)
  --database, -db TEXT     Neo4j database name (default: olink3)

Example Workflows¶

Weekly Cardiovascular Update¶

uv run python scripts/incremental_pubmed_update.py \
  --search-term '("Cardiovascular Diseases"[Mesh] OR cardiovascular OR heart) AND ("Biomarkers"[Mesh] OR "Proteomics"[Mesh] OR "protein biomarker")' \
  --days 7 \
  --database olink1

Bi-Weekly Cancer Biomarker Update¶

uv run python scripts/incremental_pubmed_update.py \
  --search-term '("Neoplasms"[Mesh] OR cancer OR tumor) AND ("Biomarkers"[Mesh] OR "Proteomics"[Mesh] OR "protein biomarker")' \
  --days 14 \
  --max-results 200 \
  --service bedrock \
  --database olink1

Daily Olink-Specific Update¶

uv run python scripts/incremental_pubmed_update.py \
  --search-term '("Olink" OR "Proximity Extension Assay") AND ("Biomarkers"[Mesh] OR protein)' \
  --days 1 \
  --max-results 50 \
  --database olink1

Output Statistics¶

The script logs comprehensive statistics:

============================================================
INCREMENTAL UPDATE COMPLETE
============================================================
Search term: cardiovascular disease protein biomarker
Time window: Last 7 days
New abstracts: 87
New entities: 342
New relationships: 156
Consolidated entities: 89
Consolidated relationships: 34
Total duration: 12.3s (12.3 min)
============================================================

Troubleshooting¶

No New Abstracts Found¶

Symptom: No new abstracts found. Database is up to date.

Causes: - All abstracts in date range already processed - PubMed query too specific (no results) - Date filter too narrow

Solutions:

# Increase time window
uv run python scripts/incremental_pubmed_update.py --search-term "..." --days 14

# Broaden search query
uv run python scripts/incremental_pubmed_update.py --search-term "protein disease"

# Check PubMed directly
# Visit: https://pubmed.ncbi.nlm.nih.gov/?term=YOUR_QUERY

Timeout During Entity Extraction¶

Symptom: Pipeline execution failed: timeout

Causes: - LLM service overloaded - Network latency - Too many abstracts

Solutions:

# Reduce batch size
uv run python scripts/incremental_pubmed_update.py --search-term "..." --max-results 50

# Use faster LLM service
uv run python scripts/incremental_pubmed_update.py --search-term "..." --service local

# Retry with same query (PMID deduplication prevents duplicates)
uv run python scripts/incremental_pubmed_update.py --search-term "..."

Duplicate Entities Created¶

Symptom: Multiple nodes for same protein/disease

Causes: - Missing UniProt/MONDO IDs - Entity consolidation not running - Synonym mismatch

Solutions:

# Run entity consolidation manually
python -m src.processors.entity_resolver --database olink1 --operation full

# Check entity coverage
python scripts/validate_synonym_coverage.py

# Enrich with canonical IDs
python enrichment_main.py --file data/uniprot_protein_dictionary.csv --database olink1

Slow Embedding Generation¶

Symptom: Step 5 takes >5 minutes

Causes: - Large number of new entities - Slow embedding model - GPU unavailable

Solutions:

# Use faster embedding model (configure in .env)
EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2

# Process in smaller batches
uv run python scripts/incremental_pubmed_update.py --search-term "..." --max-results 50

# Skip embeddings temporarily (run later)
# (Not supported yet - future enhancement)

Integration with Existing Workflows¶

After Initial Graph Creation¶

# 1. Build initial graph (one-time)
uv run python pipeline_main.py --csv-dir data/csv --pubmed-queries data/queries.txt --database olink1

# 2. Weekly updates (recurring)
uv run python scripts/incremental_pubmed_update.py --search-term "protein disease" --database olink1

With Multiple Search Terms¶

# Create update script for multiple queries
cat > scripts/weekly_update.sh << 'EOF'
#!/bin/bash
DATABASE="olink1"

uv run python scripts/incremental_pubmed_update.py --search-term "cardiovascular disease protein biomarker" --database $DATABASE
uv run python scripts/incremental_pubmed_update.py --search-term "kidney disease biomarker" --database $DATABASE
uv run python scripts/incremental_pubmed_update.py --search-term "cancer protein marker" --database $DATABASE
uv run python scripts/incremental_pubmed_update.py --search-term "diabetes biomarker" --database $DATABASE
EOF

chmod +x scripts/weekly_update.sh
./scripts/weekly_update.sh

With Community Detection¶

# 1. Run incremental update
uv run python scripts/incremental_pubmed_update.py --search-term "protein disease" --database olink1

# 2. Recompute communities (if graph structure changed significantly)
uv run python ingest_main.py --detect-communities --database olink1

# 3. Invalidate precomputed query cache
curl -X POST http://localhost:8000/v1/cache/invalidate

Best Practices¶

Run Weekly: Schedule updates every 7 days to stay current with literature
Use Specific Queries: Target specific disease areas for focused updates
Monitor Statistics: Track new abstracts/entities to detect trends
Validate Quality: Run validation scripts after large updates
Backup Database: Create backups before major updates
Invalidate Cache: Clear precomputed query cache after updates

Future Enhancements¶

[ ] Parallel processing for multiple search terms
[ ] Automatic community detection after updates
[ ] Email notifications with update statistics
[ ] Differential embedding generation (skip unchanged nodes)
[ ] Incremental validation reports
[ ] Rollback mechanism for failed updates

Knowledge Graph Creation Guide - Initial graph setup
Entity Consolidation - Deduplication algorithms
Relationship Counter - Relationship consolidation
Pipeline Main - Full pipeline for initial creation

Incremental PubMed Update Guide¶

Overview¶

Key Features¶

Quick Start¶

Basic Usage¶

Weekly Cron Job¶

How It Works¶

5-Step Pipeline¶

PMID Deduplication¶

Entity Consolidation Strategy¶

Relationship Consolidation¶

Performance Benchmarks¶

Typical Performance (100 abstracts)¶

Performance Optimization Tips¶

Command-Line Options¶

Example Workflows¶

Weekly Cardiovascular Update¶

Bi-Weekly Cancer Biomarker Update¶

Daily Olink-Specific Update¶

Output Statistics¶

Troubleshooting¶

No New Abstracts Found¶

Timeout During Entity Extraction¶

Duplicate Entities Created¶

Slow Embedding Generation¶

Integration with Existing Workflows¶

After Initial Graph Creation¶

With Multiple Search Terms¶

With Community Detection¶

Best Practices¶

Future Enhancements¶

Related Documentation¶