Skip to content

Semantic Dynamic Context Graph — Design Brief

Status: Pre-design (start fresh session from here)

Context

The Graffold currently uses a static knowledge graph (Neo4j or Memgraph): - Ingest publications/tabular data → LLM extracts entities/relationships → store as nodes/edges - User context (role, team, dataset descriptions) stored as metadata, not graph structure - No temporal dimension, no provenance chains, no statistical relationship edges - The NLDA panel collects rich user context that currently goes nowhere beyond metadata tags

The Problem

Static KGs lose signal: 1. No provenance — can't trace "this association came from row 47 of Dr. Smith's cardiology cohort" 2. No temporality — graph doesn't know when a relationship was observed or if it's still valid 3. No user context as structure — a proteomics researcher and a clinical analyst see the same flat graph 4. No statistical edges — t-test results, correlations, ANOVA outputs from the notebook are discarded 5. No cumulative understanding — each user's dataset description enriches nothing beyond that file's metadata

The Vision: Semantic Dynamic Context Graph

A living knowledge fabric where every interaction adds signal:

Core Principles

  • Nodes carry temporal validity windows, not just existence
  • Edges have provenance chains: which dataset, which user, which analysis, which row
  • User context is first-class graph structure: role, team, description → actual nodes that participate in queries
  • Statistical relationships become weighted edges with confidence intervals and sample sizes
  • The graph evolves as more users describe their datasets — descriptions enrich existing nodes
  • Context-aware queries: "show me what the proteomics team knows about IL-6" vs "show me what the clinical team knows"

Data Model Questions to Resolve

  • How do we represent temporal validity? (valid_from/valid_to on edges? versioned subgraphs?)
  • How do provenance chains compose? (dataset → row → extraction → entity → relationship)
  • How do statistical edges differ from literature-derived edges? (confidence, p-value, sample_size properties?)
  • How does user context affect query routing? (team-scoped views? role-weighted relevance?)
  • How do we handle contradictions? (Team A says protein X is upregulated, Team B says downregulated)

Architecture Questions

  • Neo4j/Memgraph temporal extensions vs. separate temporal layer?
  • How does the NLDA "Parse & Extract" output feed into graph construction?
  • How do notebook execution results (stats) become edges?
  • Event sourcing for the graph? (every mutation is an event with actor + timestamp)
  • How does this interact with the existing Glicko agent rating system?

Input Channels into the Graph

  1. Publication ingestion (existing) — PubMed, bioRxiv, PMC abstracts → entities/relationships
  2. Tabular data ingestion (existing) — CSV/Excel → chunked text → entity extraction
  3. NLDA Dataset Understanding (new) — user descriptions → parsed context → annotated nodes
  4. NLDA Notebook results (new) — statistical tests → weighted edges with provenance
  5. User feedback (existing) — thumbs up/down on query answers → confidence adjustments
  6. Cell Explorer conversations (existing) — multi-turn Q&A → implicit relationship validation

What Exists Today (code references)

  • Static KG pipeline: src/ingest_pipeline/, src/processors/
  • Entity consolidation: src/processors/entity_consolidator.py
  • Tabular ingestion: src/services/tabular_ingestion_service.py
  • Dataset description parsing: src/services/dataset_describer.py
  • User context collection: frontend/src/components/my-files/DatasetDescriptionPanel.tsx
  • Statistical analysis: frontend/src/components/my-files/PlotSuggestions.tsx (scipy in Pyodide)
  • Feedback → Glicko ratings: src/services/glicko_service.py
  • Query agents: src/agents/two_phase_agent.py, src/agents/neo4j_query_agent.py

Open Design Decisions

  • [ ] Data model for temporal + provenance edges
  • [ ] Query language extensions for context-aware traversal
  • [ ] Migration path from static KG to dynamic context graph
  • [ ] How editable table cells feed back into the graph (cell-level provenance)
  • [ ] Multi-tenant context isolation (team A's annotations vs team B's)
  • [ ] Contradiction resolution strategy
  • [ ] Performance implications of provenance chains on query latency

Next Steps

Start a new session focused on: 1. Designing the data model (property graph with temporal/provenance extensions, compatible with Neo4j and Memgraph) 2. Defining how each input channel maps to graph mutations 3. Prototyping the query semantics for context-aware traversal 4. Planning the migration from static KG to dynamic context graph