Entity Consolidation Integration Guide¶

Overview¶

The entity consolidation system provides intelligent duplicate detection and hierarchical relationship creation for disease entities using MONDO ontology integration. This guide covers the implementation, timing strategies, and configuration options.

Problem Statement¶

Knowledge graphs often contain duplicate entities with slight variations: - "Pulmonary Hypertension (PH)" vs "pulmonary hypertension (PH)" - Different IDs for the same medical concept - Missing hierarchical relationships between related diseases

Solution: Hybrid Consolidation Approach¶

When to Run Consolidation?¶

Answer: BOTH during and after database population

1. Incremental Consolidation (During Population)¶

Frequency: Every 25 chunks processed (configurable)
Scope: Name-based consolidation only (fast)
Purpose: Prevents excessive duplicate accumulation
Performance: Minimal impact on ingestion speed

2. Final Consolidation (After Population)¶

Frequency: Once at completion
Scope: Comprehensive entity resolution + ontology hierarchies
Purpose: Ensures complete data quality
Features:
Fuzzy matching (85% similarity threshold)
WHO PH Groups hierarchy creation
MONDO ontology integration

Implementation¶

Core Components¶

EntityResolver (`src/utils/entity_resolver.py`)¶

Advanced duplicate detection using fuzzy matching
MONDO ontology integration for canonical naming
WHO PH Groups hierarchy creation (Groups 1-5)
Comprehensive consolidation methods

IncrementalConsolidator (`src/utils/incremental_consolidation.py`)¶

Performance-optimized timing strategy
Configurable chunk thresholds
Balances performance with data quality

Integration Points¶

The consolidation is automatically integrated into src/pipeline/kg_pipeline.py:

# Initialization
self.entity_resolver = EntityResolver(self.db)
self.incremental_consolidator = IncrementalConsolidator(
    db=self.db,
    chunk_threshold=25  # Run every 25 chunks
)

# During batch processing
for chunk_data in batch:
    consolidation_stats = self.incremental_consolidator.on_chunk_processed(chunk_data["chunk_id"])
    if consolidation_stats:
        logger.info(f"Incremental consolidation completed: {consolidation_stats}")

# Final consolidation
consolidation_stats = self.incremental_consolidator.final_consolidation()
logger.info(f"Consolidation results: {consolidation_stats}")

Configuration Options¶

Chunk Thresholds¶

# More frequent consolidation (every 10 chunks)
IncrementalConsolidator(db=db, chunk_threshold=10)

# Less frequent consolidation (every 50 chunks)
IncrementalConsolidator(db=db, chunk_threshold=50)

# Disable incremental (batch-only approach)
# Simply don't call on_chunk_processed() in the loop

Consolidation Strategies¶

Strategy	Chunk Threshold	Performance	Duplicate Control	Use Case
Real-time	1	❌ Slow	✅ Excellent	Small datasets
Hybrid (Recommended)	25-50	✅ Balanced	✅ Good	Production
Batch-only	Disabled	✅ Fast	❌ Poor	Development

Results¶

Consolidation Statistics¶

From testing on cvd1 database: - 64 duplicate nodes merged - 8 WHO PH hierarchy relationships created - 26,284 MONDO ontology entries loaded - 85% fuzzy matching threshold applied

Example Consolidations¶

"pulmonary hypertension (PH)" → "Pulmonary Hypertension (PH)"
Multiple disease variants consolidated to canonical forms
Hierarchical relationships: "Pulmonary Hypertension (PH)" → WHO Groups 1-5

WHO PH Groups Hierarchy¶

The system automatically creates ontological relationships: 1. Group 1: Pulmonary Arterial Hypertension (PAH) 2. Group 2: PH due to Left Heart Disease
3. Group 3: PH due to Lung Disease 4. Group 4: Chronic Thromboembolic PH (CTEPH) 5. Group 5: PH with Unclear/Multifactorial Mechanisms

Performance Characteristics¶

Incremental Consolidation¶

Time: ~2-5 seconds per run
Scope: Name-based matching only
Database Impact: Minimal
Memory Usage: Low

Final Consolidation¶

Time: ~30-60 seconds (depending on database size)
Scope: Full fuzzy matching + hierarchy creation
Database Impact: Moderate (runs once)
Memory Usage: Higher (loads full MONDO ontology)

Example Workflow¶

Processing 100 chunks:
  Chunks 1-24:   Normal ingestion
  Chunk 25:      Incremental consolidation triggered ⚡
  Chunks 26-49:  Normal ingestion  
  Chunk 50:      Incremental consolidation triggered ⚡
  Chunks 51-74:  Normal ingestion
  Chunk 75:      Incremental consolidation triggered ⚡
  Chunks 76-99:  Normal ingestion
  Chunk 100:     Final comprehensive consolidation 🎯

Query Examples¶

After consolidation, you can query the enhanced data:

// Find consolidated entities by frequency
MATCH ()-[r]->() 
WHERE r.frequency > 1 
RETURN r.pattern_id, r.frequency
ORDER BY r.frequency DESC

// Find WHO PH hierarchy
MATCH (parent)-[r:ONTOLOGY]->(child)
WHERE parent.name CONTAINS 'Pulmonary Hypertension'
RETURN parent.name, r.type, child.name

// Find duplicate patterns that were merged
MATCH (n:Disease) 
WHERE n.consolidated_from IS NOT NULL
RETURN n.name, n.consolidated_from

Testing¶

Run the consolidated test suite:

# Run all tests
cd /Users/apple/Developer/graffold/graffold-api
python tests/run_tests.py --database cvd1

# Run only relationship analysis
python tests/run_tests.py --test relationships --database cvd1

# Run relationship analysis with LLM-based type consolidation (dry run)
python tests/run_tests.py --test relationships --consolidate-types --database cvd1

# Execute LLM-based type consolidation (live changes)
python tests/run_tests.py --test relationships --consolidate-types --execute --database cvd1

# Run only consolidation tests  
python tests/run_tests.py --test consolidation --database cvd1

Individual test components:

# Unified relationship analysis with type consolidation
python tests/test_relationship_analysis.py cvd1 --consolidate-types

# Standalone LLM-based relationship type consolidation
python src/utils/relationship_type_consolidator.py --database cvd1 --dry-run

# Execute relationship type consolidation
python src/utils/relationship_type_consolidator.py --database cvd1 --execute

# Test LLM consolidation with sample data
python test_llm_consolidation.py

# Unified consolidation tests
python tests/test_consolidation_unified.py cvd1

# Run with pytest
pytest tests/test_relationship_analysis.py
pytest tests/test_consolidation_unified.py

Monitoring¶

The system provides detailed logging: - Consolidation statistics after each run - Performance metrics (timing, entities processed) - Error handling and recovery - Progress tracking during batch processing

Best Practices¶

Use hybrid approach for production workloads
Monitor consolidation logs for performance tuning
Adjust chunk thresholds based on dataset size
Run final consolidation always for data quality
Test on smaller datasets before full deployment

Troubleshooting¶

Common Issues¶

High memory usage: Reduce chunk threshold
Slow performance: Increase chunk threshold
Missing hierarchies: Ensure final consolidation runs
Over-consolidation: Adjust fuzzy matching threshold

Debug Commands¶

# Check consolidation status (works with both Neo4j and Memgraph)
python -c "from src.processors.entity_resolver import EntityResolver; from src.core.database import Neo4jDatabase; db = Neo4jDatabase('cvd1'); resolver = EntityResolver(db); print(resolver.get_consolidation_stats())"

# Manual consolidation
python -c "from src.processors.entity_resolver import EntityResolver; from src.core.database import Neo4jDatabase; db = Neo4jDatabase('cvd1'); resolver = EntityResolver(db); stats = resolver.run_full_entity_resolution(); print(stats)"

LLM-Based Relationship Type Consolidation¶

Overview¶

The system now includes intelligent relationship type consolidation using Large Language Models (LLMs). This automatically identifies and merges semantically similar relationship types without requiring hardcoded mappings.

Features¶

Semantic Analysis¶

LLM-Powered: Uses language models to understand semantic similarity
Context-Aware: Considers biological/medical context
Flexible: No hardcoded rules - adapts to any domain
Validation: LLM reviews consolidation plans before execution

Example Consolidations¶

The system can identify and consolidate relationship types like: - "associated_with", "related_to", "affiliated_with" → "ASSOCIATED_WITH" - "causes", "leads_to", "results_in", "induces" → "CAUSES" - "treats", "therapeutic_for", "cures" → "TREATS"

Safety Features¶

Dry Run Mode: Test consolidations without making changes
LLM Validation: AI reviews plans before execution
Metadata Preservation: Original types stored as metadata
Rollback Support: Changes can be tracked and reversed

Process Flow¶

Discovery: Extract all relationship types from database
Semantic Analysis: LLM analyzes types for similarity
Grouping: Similar types grouped with canonical names
Plan Creation: Consolidation plan with frequency analysis
Validation: LLM reviews plan for appropriateness
Execution: Apply consolidations (dry run or live)

Configuration¶

LLM Selection¶

# Use local LLM (default)
consolidator = RelationshipTypeConsolidator(llm_service="local")

# Use OpenAI
consolidator = RelationshipTypeConsolidator(llm_service="openai")

# Use SageMaker
consolidator = RelationshipTypeConsolidator(llm_service="sagemaker")

Execution Modes¶

# Dry run analysis (safe)
python src/utils/relationship_type_consolidator.py --database cvd1 --dry-run

# Live execution
python src/utils/relationship_type_consolidator.py --database cvd1 --execute

Integration with Analysis Pipeline¶

The relationship type consolidation is integrated into the unified relationship analyzer:

from tests.test_relationship_analysis import UnifiedRelationshipAnalyzer

analyzer = UnifiedRelationshipAnalyzer(database="cvd1")

# Analyze includes type consolidation recommendations
analysis = analyzer.analyze_relationship_frequencies()
print(analysis["type_consolidation"])

# Execute consolidation
results = analyzer.consolidate_relationship_types(dry_run=True)

Query Examples¶

After consolidation, query consolidated relationships:

// Find relationships with consolidation metadata
MATCH ()-[r]->() 
WHERE r.consolidated = true
RETURN r.original_type, r.canonical_type, count(r)
ORDER BY count(r) DESC

// Query by canonical type
MATCH ()-[r]->() 
WHERE r.canonical_type = 'ASSOCIATED_WITH'
RETURN count(r) as total_associations

// Find consolidation patterns
MATCH ()-[r]->() 
WHERE r.pattern_id IS NOT NULL
RETURN r.pattern_id, r.canonical_type, count(r)
ORDER BY count(r) DESC

Entity Consolidation Integration Guide¶

Overview¶

Problem Statement¶

Solution: Hybrid Consolidation Approach¶

When to Run Consolidation?¶

1. Incremental Consolidation (During Population)¶

2. Final Consolidation (After Population)¶

Implementation¶

Core Components¶

EntityResolver (src/utils/entity_resolver.py)¶

IncrementalConsolidator (src/utils/incremental_consolidation.py)¶

Integration Points¶

Configuration Options¶

Chunk Thresholds¶

Consolidation Strategies¶

Results¶

Consolidation Statistics¶

Example Consolidations¶

WHO PH Groups Hierarchy¶

Performance Characteristics¶

Incremental Consolidation¶

Final Consolidation¶

Example Workflow¶

Query Examples¶

Testing¶

Monitoring¶

Best Practices¶

Troubleshooting¶

Common Issues¶

Debug Commands¶

LLM-Based Relationship Type Consolidation¶

Overview¶

Features¶

Semantic Analysis¶

Example Consolidations¶

Safety Features¶

Process Flow¶

Configuration¶

LLM Selection¶

Execution Modes¶

Integration with Analysis Pipeline¶

Query Examples¶

EntityResolver (`src/utils/entity_resolver.py`)¶

IncrementalConsolidator (`src/utils/incremental_consolidation.py`)¶