Metrics Dashboard

Comprehensive metrics tracking for agent query intelligence and incremental pipeline performance.

Overview

The metrics dashboard tracks key performance indicators across: - Agent Ratings: Glicko-based leaderboard, win rates, trend direction, confidence intervals - Query Intelligence: Multi-hop traversal, community summaries, mode classification, entity disambiguation - Performance: Cache hit rates, query latency percentiles (across Neo4j and Memgraph backends) - Cost Tracking: LLM token usage and estimated costs by operation - Feedback Analytics: Satisfaction rate, thumbs up/down, star ratings - Eval Insights: Query success/error rates, follow-up rate, mode distribution - CI Results: Latest eval and E2E test summaries - Pipeline Operations: Incremental update statistics

API Endpoints

All endpoints require authentication via Authorization: Bearer <token> header.

Comprehensive Dashboard

GET /v1/metrics

Returns all query intelligence metrics in a single response (see below for full shape).

Unified Dashboard Overview

GET /v1/dashboard/overview

Returns a single response combining agent ratings, query performance, costs, feedback, eval analytics, CI results, and incremental update stats:

{
  "agent_ratings": {
    "total_agents": 3,
    "total_games": 120,
    "average_rating": 1542.5,
    "top_agents": [
      {"agent_id": "two_phase", "rating": 1580.2, "win_rate": 0.78, "games_played": 50},
      {"agent_id": "neo4j", "rating": 1545.1, "win_rate": 0.72, "games_played": 40}
    ]
  },
  "query_performance": {
    "latency": {"total_queries": 500, "p50_ms": 250.0, "p95_ms": 1200.0, "p99_ms": 2500.0, "avg_ms": 450.0},
    "cache": {"total_queries": 500, "hit_rate": 0.30},
    "multi_hop": {"total_queries": 150, "success_rate": 0.947},
    "disambiguation": {"total_queries": 300, "avg_recall": 0.90},
    "community_usage": {"total_queries": 200, "usage_rate": 0.90},
    "mode_classification": {"total_queries": 500, "auto_classified_rate": 0.85, "mode_distribution": {"local": 200, "global": 150}}
  },
  "cost_summary": {
    "total_records": 1200,
    "total_prompt_tokens": 500000,
    "total_completion_tokens": 150000,
    "total_cost": 12.50,
    "cost_by_operation": {"query": 8.0, "extraction": 3.5, "embedding": 1.0}
  },
  "feedback": {
    "total_feedback": 80,
    "thumbs_up": 60,
    "thumbs_down": 15,
    "average_star_rating": 3.8,
    "satisfaction_rate": 0.80
  },
  "eval": {
    "total_records": 500,
    "success_rate": 0.96,
    "avg_latency_ms": 450.0,
    "error_rate": 0.04,
    "follow_up_rate": 0.12,
    "mode_distribution": {"local": 200, "global": 150, "hybrid": 100, "naive": 50}
  },
  "ci": {
    "eval": {"entity_recall_avg": 0.85, "retrieval_recall_avg": 0.82},
    "e2e": {"passed": true, "pass_rate": 1.0}
  },
  "incremental_updates": {
    "total_updates": 10,
    "success_rate": 0.90,
    "avg_entity_merge_rate": 0.25
  }
}

Agent Comparison

GET /v1/dashboard/agent-comparison

Side-by-side comparison of all agents with ratings, trends, and confidence intervals:

{
  "agents": [
    {
      "agent_id": "two_phase",
      "rating": 1580.2,
      "rating_deviation": 45.3,
      "confidence_interval": {"lower": 1491.4, "upper": 1669.0},
      "win_rate": 0.78,
      "games_played": 50,
      "trend_direction": 25.4,
      "sufficient_data": true
    }
  ]
}

Query Intelligence Metrics (GET /v1/metrics)

{
  "multi_hop": {
    "total_queries": 150,
    "successful_queries": 142,
    "success_rate": 0.947,
    "timeout_count": 8,
    "timeout_rate": 0.053,
    "fallback_count": 8,
    "fallback_rate": 0.053
  },
  "community_usage": {
    "total_queries": 200,
    "queries_with_summaries": 180,
    "usage_rate": 0.90,
    "total_summaries_retrieved": 900,
    "avg_summaries_per_query": 4.5
  },
  "mode_classification": {
    "total_queries": 500,
    "auto_classified_count": 425,
    "auto_classified_rate": 0.85,
    "explicit_mode_count": 75,
    "mode_distribution": {
      "local": 200,
      "global": 150,
      "hybrid": 100,
      "naive": 50
    }
  },
  "disambiguation": {
    "total_queries": 300,
    "total_requested": 1500,
    "total_resolved": 1350,
    "avg_recall": 0.90
  },
  "cache": {
    "total_queries": 500,
    "cache_hits": 150,
    "cache_misses": 350,
    "hit_rate": 0.30
  },
  "latency": {
    "total_queries": 500,
    "p50_ms": 250.0,
    "p95_ms": 1200.0,
    "p99_ms": 2500.0,
    "avg_ms": 450.0,
    "min_ms": 50.0,
    "max_ms": 5000.0
  },
  "incremental_updates": {
    "total_updates": 10,
    "successful_updates": 9,
    "failed_updates": 1,
    "success_rate": 0.90,
    "avg_duration_seconds": 300.0,
    "avg_new_abstracts": 100.0,
    "avg_entity_merge_rate": 0.25,
    "avg_relationship_merge_rate": 0.30,
    "total_new_abstracts": 1000,
    "total_new_entities": 2000,
    "total_merged_entities": 500,
    "total_new_relationships": 3000,
    "total_merged_relationships": 900
  }
}

Individual Metric Endpoints

Multi-Hop Query Success Rate

GET /v1/metrics/multi-hop

Tracks success rate, timeout rate, and fallback rate for multi-hop queries (hop_depth > 1).

Key Metrics: - success_rate: Percentage of multi-hop queries that completed successfully - timeout_rate: Percentage of queries that hit the 5s timeout - fallback_rate: Percentage of queries that used fallback strategy (reduced seed set)

Community Summary Usage

GET /v1/metrics/community

Tracks usage of community summaries in global/hybrid/mix queries.

Key Metrics: - usage_rate: Percentage of eligible queries that retrieved community summaries - avg_summaries_per_query: Average number of summaries retrieved per query

Query Mode Classification

GET /v1/metrics/classification

Tracks auto-classification vs explicit mode selection.

Key Metrics: - auto_classified_rate: Percentage of queries that used auto-classification - mode_distribution: Breakdown of queries by mode (local, global, hybrid, naive)

Note: True classification accuracy requires manual labels. This endpoint reports auto-classification rate and mode distribution.

Entity Disambiguation Recall

GET /v1/metrics/disambiguation

Tracks entity resolution success rate.

Key Metrics: - avg_recall: Average percentage of requested entities successfully resolved to canonical IDs - total_requested: Total entity mentions in queries - total_resolved: Total entities resolved to UniProt/MONDO IDs

Cache Hit Rate

GET /v1/metrics/cache (deprecated - use /v1/cache/stats)

Tracks precomputed query cache effectiveness.

Key Metrics: - hit_rate: Percentage of queries served from cache - cache_hits: Number of cache hits - cache_misses: Number of cache misses

Note: Use /v1/cache/stats for more detailed cache statistics.

Query Latency Percentiles

GET /v1/metrics/latency

Tracks query execution time distribution.

Key Metrics: - p50_ms: Median query latency (50th percentile) - p95_ms: 95th percentile latency (SLA target) - p99_ms: 99th percentile latency (outlier detection) - avg_ms: Average query latency - min_ms / max_ms: Latency range

Incremental Update Metrics

GET /v1/metrics/incremental-updates

Tracks incremental pipeline update performance.

Key Metrics: - success_rate: Percentage of successful pipeline runs - avg_duration_seconds: Average pipeline execution time - avg_entity_merge_rate: Average percentage of entities merged (vs created) - avg_relationship_merge_rate: Average percentage of relationships merged (vs created)

Integration

Query Service Integration

The MetricsTracker is automatically integrated into the query execution flow:

# Metrics are recorded automatically during query execution
result = query_service.execute_query(
    agent=agent,
    question="What proteins are associated with heart disease?",
    mode="local",
    hop_depth=2,
    session_id="sess_123",
)

# Metrics include:
# - Mode classification (auto vs explicit)
# - Query latency
# - Multi-hop parameters (hop_depth, timeout, fallback)
# - Community summary retrieval count
# - Entity disambiguation results
# - Cache hit/miss

Incremental Pipeline Integration

Record metrics for incremental pipeline updates:

from src.services.metrics_tracker import MetricsTracker

tracker = MetricsTracker()

# Record incremental update metrics
tracker.record_incremental_update(
    pipeline_run_id="run_20260225_001",
    duration_seconds=300.5,
    new_abstracts=100,
    existing_pmids_skipped=50,
    new_entities=200,
    merged_entities=50,
    new_relationships=300,
    merged_relationships=100,
    new_embeddings=250,
    success=True,
)

Usage Examples

Python Client

import requests

BASE = "http://localhost:8000"
HEADERS = {"Authorization": "Bearer your_token"}

# Get comprehensive dashboard
response = requests.get(f"{BASE}/v1/metrics", headers=HEADERS)
metrics = response.json()

print(f"Multi-hop success rate: {metrics['multi_hop']['success_rate']:.1%}")
print(f"Cache hit rate: {metrics['cache']['hit_rate']:.1%}")
print(f"P95 latency: {metrics['latency']['p95_ms']:.0f}ms")

# Get specific metric
response = requests.get(f"{BASE}/v1/metrics/latency", headers=HEADERS)
latency = response.json()
print(f"Median latency: {latency['p50_ms']:.0f}ms")

cURL

# Comprehensive dashboard
curl -H "Authorization: Bearer $API_AUTH_TOKEN" \
  http://localhost:8000/v1/metrics

# Multi-hop metrics
curl -H "Authorization: Bearer $API_AUTH_TOKEN" \
  http://localhost:8000/v1/metrics/multi-hop

# Latency percentiles
curl -H "Authorization: Bearer $API_AUTH_TOKEN" \
  http://localhost:8000/v1/metrics/latency

Monitoring & Alerting

Key Performance Indicators (KPIs)

Query Intelligence: - Multi-hop success rate > 95% - Community usage rate > 80% (for global queries) - Entity disambiguation recall > 90%

Performance: - Cache hit rate > 30% - P95 latency < 2000ms - P99 latency < 5000ms

Pipeline Operations: - Incremental update success rate > 95% - Average update duration < 15 minutes (for 100 abstracts) - Entity merge rate 20-40% (indicates effective consolidation)

Alert Thresholds

# Example alert logic
metrics = get_metrics_dashboard()

# Alert: High multi-hop timeout rate
if metrics["multi_hop"]["timeout_rate"] > 0.10:
    alert("Multi-hop timeout rate exceeds 10%")

# Alert: Low cache hit rate
if metrics["cache"]["hit_rate"] < 0.20:
    alert("Cache hit rate below 20% - consider precomputing more queries")

# Alert: High P95 latency
if metrics["latency"]["p95_ms"] > 2000:
    alert("P95 latency exceeds 2000ms - investigate slow queries")

# Alert: Low entity merge rate
if metrics["incremental_updates"]["avg_entity_merge_rate"] < 0.15:
    alert("Entity merge rate below 15% - check consolidation logic")

Architecture

Metrics Collection Flow

Query Request
    ↓
QueryService.execute_query()
    ↓
[Cache Check] → MetricsTracker.record_query(cache_hit=True)
    ↓
QueryModeRouter.route()
    ↓
Agent.query()
    ↓
MetricsTracker.record_query(
    mode_used, latency_ms, hop_depth,
    community_summaries, entities_resolved, etc.
)
    ↓
Query Response

Data Storage

• In-Memory: Metrics stored in MetricsTracker._query_metrics and MetricsTracker._update_metrics lists
• Thread-Safe: All operations protected by threading.Lock
• Persistence: Metrics are ephemeral (reset on server restart)

Future Enhancement: Add Redis persistence for metrics across server restarts.

Performance Impact

• Minimal Overhead: Metrics recording adds <1ms per query
• Memory Usage: ~1KB per query metric, ~2KB per update metric
• Scalability: Suitable for 10K-100K queries before memory concerns

Troubleshooting

No Metrics Returned

Symptom: All endpoints return zero counts.

Cause: No queries have been executed since server startup.

Solution: Execute some queries to populate metrics.

Inaccurate Classification Accuracy

Symptom: mode_classification.auto_classified_rate doesn't match expectations.

Cause: This metric reports auto-classification rate, not accuracy. True accuracy requires manual labels.

Solution: Use data/test_queries.yaml with manual labels for accuracy evaluation.

Missing Incremental Update Metrics

Symptom: incremental_updates.total_updates is zero.

Cause: Incremental pipeline updates must explicitly call tracker.record_incremental_update().

Solution: Integrate metrics recording into scripts/incremental_pubmed_update.py or pipeline_main.py.

Related Documentation

• Precomputed Query Cache - Cache hit rate optimization
• Multi-Hop Traversal - Multi-hop query implementation
• Community Detection - Community summary validation
• Incremental PubMed Update - Pipeline metrics integration