Skip to content

Grafana Dashboard — Eval & E2E Metrics

Quick Start

# Start the monitoring stack (Prometheus + Pushgateway + Grafana)
docker-compose --profile monitoring up -d

# Open Grafana
open http://localhost:3000   # admin / admin (or $GRAFANA_ADMIN_PASSWORD)

The Graffold — Eval & E2E dashboard is auto-provisioned at startup.

Dashboard URL: http://localhost:3000/d/graffold-eval-e2e

Architecture

CI Workflows ──POST──▶ Pushgateway:9091 ◀──scrape── Prometheus:9090 ◀──query── Grafana:3000
                         (eval.yml)                                              │
                         (e2e.yml)                                          Dashboards
                                                                            + Alerts
  • eval.yml pushes eval_* metrics (entity_recall, retrieval_recall, classification_accuracy, latency percentiles) per config and category via reporter.push_to_grafana().
  • e2e.yml pushes e2e_passed, e2e_run_timestamp_seconds, and e2e_a11y_violations_total.
  • Prometheus scrapes Pushgateway every 15s.
  • Grafana queries Prometheus and renders panels.

Dashboard Panels

Eval Panels (rows 1–3)

Panel Metric Description
Answer Relevance eval_entity_recall Expected entities found in answer text
Retrieval Recall eval_retrieval_recall Expected entities found in retrieved graph nodes
Mode Classification Accuracy eval_classification_accuracy Predicted vs expected query mode
Latency p50/p95/p99 eval_p50_s, eval_p95_s, eval_p99_s Query latency percentiles

Per-Category Breakdown (row 3)

Same metrics filtered by category label (local, global, multi-hop, hybrid, disambiguation). Use the $category template variable to filter.

E2E Panels (row 4)

Panel Metric Description
Journey Pass/Fail e2e_passed 1 = pass, 0 = fail (stat panel with color mapping)
Execution Time Trend e2e_run_timestamp_seconds When E2E runs occurred
Accessibility Violations e2e_a11y_violations_total axe-core violation count per run

Alerts

Provisioned alert rules fire when any eval metric drops >5% compared to the previous measurement:

  • eval_entity_recall drop >5%
  • eval_retrieval_recall drop >5%
  • eval_classification_accuracy drop >5%

Alert rules are defined in grafana/provisioning/alerting/eval_alerts.yml. Configure notification channels (Slack, email, PagerDuty) in Grafana UI under Alerting → Contact points.

Configuration

Environment Variables

Variable Default Description
GRAFANA_ADMIN_PASSWORD admin Grafana admin password
PUSHGATEWAY_URL — Set in GitHub repo variables for CI metric push

CI Integration

Both eval.yml and e2e.yml push metrics when PUSHGATEWAY_URL is set as a GitHub repository variable. For self-hosted runners, point it at your Pushgateway instance.

File Layout

grafana/
├── prometheus.yml                          # Prometheus scrape config
├── dashboards/
│   └── eval-e2e.json                       # Dashboard definition
└── provisioning/
    ├── datasources/prometheus.yml          # Auto-configure Prometheus datasource
    ├── dashboards/dashboards.yml           # Auto-load dashboard JSON
    └── alerting/eval_alerts.yml            # Alert rules for metric drops

Local Development

Push test metrics manually to verify the dashboard:

# Push a sample eval metric
cat <<EOF | curl --data-binary @- http://localhost:9091/metrics/job/eval_harness/config_label/Standard_bedrock_local
eval_entity_recall{config="Standard_bedrock_local"} 0.85
eval_retrieval_recall{config="Standard_bedrock_local"} 0.72
eval_classification_accuracy{config="Standard_bedrock_local"} 0.91
eval_p50_s{config="Standard_bedrock_local"} 0.25
eval_p95_s{config="Standard_bedrock_local"} 1.2
eval_p99_s{config="Standard_bedrock_local"} 2.5
EOF

# Push a sample E2E metric
cat <<EOF | curl --data-binary @- http://localhost:9091/metrics/job/e2e_ci/instance/test
e2e_passed 1
e2e_run_timestamp_seconds $(date +%s)
e2e_a11y_violations_total 0
EOF

Then open http://localhost:3000/d/graffold-eval-e2e to see the data.