Hetzner Operations Guide¶

Architecture¶

┌─────────────────────────────────────────────────┐
│  Hetzner VPS                                    │
│                                                 │
│  ┌──────────┐  ┌──────────┐  ┌──────────────┐  │
│  │ FastAPI   │  │  Redis   │  │ Neo4j or     │  │
│  │ (granian) │──│  :6379   │  │ Memgraph     │  │
│  │  :8000    │  └──────────┘  │   :7687      │  │
│  └──────────┘                 └──────────────┘  │
│       │                                         │
│  Dockerfile.hetzner                             │
│  docker-compose.yml (profile: api)              │
└─────────────────────────────────────────────────┘
         │
    GitHub Actions (push to main)
         │
    rsync → docker build → docker compose up

Memgraph dev deployment: For development or lightweight deployments, use the memgraph Docker Compose profile instead of a standalone Neo4j instance:
docker compose -f docker-compose.yml -f docker-compose.memgraph.yml --profile memgraph up -d
Set DATABASE_TYPE=memgraph in .env when using this profile.

Server Layout¶

/opt/graffold/
├── app/                    # rsync'd from GitHub (graffold-api repo)
│   ├── Dockerfile.hetzner  # Production build (Python 3.13 + spacy)
│   ├── docker-compose.yml  # Service orchestration
│   ├── .env                # Server-specific secrets (NOT synced from GitHub)
│   ├── litegraf/           # Cloned separately on server (excluded from rsync)
│   ├── api/
│   ├── src/
│   ├── cli/
│   └── ...
└── (docker volumes)        # graph-db-data, redis-data, etc.

Automatic Deployment (GitHub Actions)¶

Every push to main triggers .github/workflows/deploy.yml:

CI tests run first (lint + unit/integration/property/slow tests)
Code is rsync'd to /opt/graffold/app/ (excludes .env, litegraf/, .venv)
Docker image rebuilt from Dockerfile.hetzner
fastapi service restarted via docker compose up -d --no-deps --force-recreate fastapi
Health check polls http://localhost:8000/health for up to 60s
Post-deploy prints container status and disk usage

Required GitHub Secrets¶

Secret	Description
`HETZNER_SSH_KEY`	Private SSH key for `root@<server>`
`HETZNER_HOST`	Server IP or hostname

Set these in GitHub → Settings → Secrets and variables → Actions.

Manual Deployment¶

SSH into the server and run:

ssh root@<HETZNER_HOST>
cd /opt/graffold/app

# Pull latest code (if not using rsync)
git pull origin main

# Rebuild and restart API only (preserves Redis/Neo4j data)
docker build -f Dockerfile.hetzner -t graffold-api:latest .
docker compose up -d --no-deps --force-recreate fastapi

# Watch logs
docker compose logs -f fastapi

Common Operations¶

Check service status¶

COMPOSE="docker compose -f docker-compose.yml -f docker-compose.hetzner.yml --profile api"
$COMPOSE ps
$COMPOSE logs fastapi --tail 50
$COMPOSE logs redis --tail 20

Restart a single service (no rebuild)¶

COMPOSE="docker compose -f docker-compose.yml -f docker-compose.hetzner.yml --profile api"
$COMPOSE restart fastapi
$COMPOSE restart redis

Full rebuild (after dependency changes)¶

COMPOSE="docker compose -f docker-compose.yml -f docker-compose.hetzner.yml --profile api"
$COMPOSE build --no-cache fastapi
$COMPOSE up -d --no-deps --force-recreate fastapi

Update litegraf on server¶

litegraf is excluded from rsync and lives separately on the server:

cd /opt/graffold/app/litegraf
git pull origin main
# Rebuild API image (litegraf is COPY'd into the Docker image)
cd /opt/graffold/app
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.hetzner.yml --profile api"
$COMPOSE build fastapi
$COMPOSE up -d --no-deps --force-recreate fastapi

Update .env on server¶

The .env file is server-specific and never overwritten by deploys:

vim /opt/graffold/app/.env
docker compose restart fastapi

View resource usage¶

docker stats --no-stream
df -h /
docker system df

Clean up Docker disk space¶

docker system prune -f          # remove stopped containers, unused networks
docker builder prune -f         # remove build cache
docker image prune -a -f        # remove all unused images (aggressive)

Graph database operations¶

Neo4j:

# Access Neo4j browser
# Forward port if needed: ssh -L 7474:localhost:7474 root@<HOST>
# Then open http://localhost:7474

# Backup Neo4j data
docker compose exec neo4j neo4j-admin database dump neo4j --to-path=/data/backups/
docker cp graffold-neo4j:/data/backups/ ./neo4j-backup-$(date +%Y%m%d)/

Memgraph:

# Access Memgraph Lab (if running)
# Forward port if needed: ssh -L 3000:localhost:3000 root@<HOST>

Redis operations¶

# Check Redis memory
docker compose exec redis redis-cli INFO memory | grep used_memory_human

# Flush all caches (sessions will be lost)
docker compose exec redis redis-cli FLUSHALL

Rollback¶

If a deploy breaks the API:

ssh root@<HETZNER_HOST>
cd /opt/graffold/app

# Option 1: Revert to previous Docker image layer
docker compose logs fastapi --tail 50   # check what went wrong

# Option 2: Revert code and rebuild
git log --oneline -5                    # find the last good commit
git checkout <good-commit-sha>
docker build -f Dockerfile.hetzner -t graffold-api:latest .
docker compose up -d --no-deps --force-recreate fastapi

First-Time Server Setup¶

If setting up a fresh Hetzner VPS:

# 1. Install Docker
curl -fsSL https://get.docker.com | sh
systemctl enable docker

# 2. Create app directory
mkdir -p /opt/graffold/app

# 3. Clone repos
cd /opt/graffold/app
git clone <graffold-api-repo-url> .
git clone <litegraf-repo-url> litegraf

# 4. Create .env from template
cp .env.example .env
vim .env   # fill in real values (database password, AWS keys, API tokens, etc.)

# 5. Start services
COMPOSE="docker compose -f docker-compose.yml -f docker-compose.hetzner.yml --profile api"
$COMPOSE up -d

# 6. Verify
curl http://localhost:8000/health

Monitoring¶

The API exposes these health endpoints:

Endpoint	Purpose
`GET /health`	Overall health (graph database + Redis connectivity)
`GET /health/live`	Liveness probe (is the process running?)
`GET /health/ready`	Readiness probe (can it serve requests?)

For Prometheus/Grafana monitoring, start with the monitoring profile:

docker compose --profile monitoring up -d
# Grafana at :3000, Prometheus at :9090, Pushgateway at :9091

Troubleshooting¶

Symptom	Likely cause	Fix
API returns 503	Redis not running	`docker compose restart redis`
Slow first request	spacy model loading	Wait ~30s after container start
`ModuleNotFoundError`	litegraf not in image	Rebuild: `docker build -f Dockerfile.hetzner ...`
Disk full	Docker images/layers	`docker system prune -a -f`
Neo4j/Memgraph/FalkorDB connection refused	Database not started or wrong URI	Check `.env` NEO4J_URI and DATABASE_TYPE, `docker compose ps`
Health check timeout in CI	Server under load or slow boot	Check `docker compose logs fastapi` on server

Dev Environment (Memgraph)¶

The dev environment at dev.graffold.com runs Memgraph instead of Neo4j:

# Deploy to dev
docker compose -f docker-compose.hetzner-dev.yml up -d

Set DATABASE_TYPE=memgraph in .env.dev. The API automatically uses the correct driver and query syntax for any supported backend (neo4j, memgraph, falkordb).