Part I - JILHQ (Dedicated Server)
1. Source Files
JILHQ runs on a dedicated Hetzner CPX52 server at 167.235.150.16 (hq.jilsovereign.com) and serves as the fleet management server. It registers images, signs digests, and orchestrates distribution to all validators.
| File | Purpose |
services/jilhq/src/index.ts | Main Express server (~3200 lines). Distribution endpoints, agent registration, dynamic endpoint lookup |
services/jilhq/migrations/009_distribution_jobs.sql | Creates hq_distribution_jobs table |
deploy/hetzner/docker-compose.hq.yml | JILHQ service config, SIGNING_SECRET env var |
deploy-to-hetzner.sh | Added --distribute flag, JILHQ_URL/LOCAL_REGISTRY vars, distribution step [5/5] |
2. Database Tables
18 tables across 9 migrations, all residing in JILHQ's PostgreSQL instance (jil-postgres).
| # | Table | Migration | Purpose |
| 1 | hq_nodes | 001 (+007,+008) | Fleet node registry with agent URLs |
| 2 | hq_images | 001 | Container image registry with signatures |
| 3 | hq_promotions | 001 | Image track promotions (devnet→testnet→mainnet) |
| 4 | hq_certs | 001 | mTLS certificate management |
| 5 | hq_audit_log | 001 | Immutable audit ledger |
| 6 | hq_configs | 001 | Fleet configuration push per jurisdiction |
| 7 | hq_alert_rules | 002 | Alert rule definitions |
| 8 | hq_alerts | 002 | Fired alert instances |
| 9 | hq_validator_keys | 003 | Validator signing key registry |
| 10 | hq_image_deployments | 003 | Per-node image deployment tracking |
| 11 | hq_auth_decisions | 003 | Authorization decision log |
| 12 | hq_quorum_config | 004 | Per-zone quorum configuration |
| 13 | hq_consensus_rounds | 005 (+008) | Settlement consensus rounds |
| 14 | hq_consensus_signatures | 005 (+006) | Per-validator consensus signatures |
| 15 | hq_validator_ed25519_keys | 006 | Ed25519 key registry |
| 16 | hq_equivocation_evidence | 007 | Double-signing evidence |
| 17 | hq_validator_heartbeats | 008 | Heartbeat history |
| 18 | hq_distribution_jobs | 009 | Image distribution job tracking |
Key distribution tables: hq_nodes (stores agent_url in metadata), hq_images (signed digests), hq_image_deployments (per-node status), hq_distribution_jobs (job tracking), hq_audit_log (all actions logged).
3. REST Endpoints
Distribution Endpoints (new)
| Method | Path | Purpose |
| POST | /v1/fleet/distribute | Distribute signed image to validators |
| GET | /v1/fleet/distribute/:jobId | Poll distribution job status |
| GET | /v1/fleet/distributions | List recent distribution jobs (limit 50) |
| POST | /v1/fleet/nodes/:nodeId/register-agent | Register/update a validator's agent URL |
| GET | /v1/fleet/agents | List all registered agent endpoints |
Image Registry Endpoints (existing)
| Method | Path | Purpose |
| GET | /v1/registry/images | List all images (filter by ?track=) |
| POST | /v1/registry/images | Register new image with digest |
| POST | /v1/registry/images/:name/sign | Sign image digest (HMAC-SHA256) |
| POST | /v1/registry/images/:name/verify | Verify image signature |
| POST | /v1/registry/images/:name/promote | Promote devnet→testnet→mainnet |
Fleet Endpoints (existing)
| Method | Path | Purpose |
| POST | /v1/fleet/push-image | Record deployment intent |
| POST | /v1/fleet/nodes/:nodeId/verify-deployment | Verify running digest matches signed |
| GET | /v1/fleet/deployments | Deployment status across all nodes |
4. Environment Variables
| Variable | Default | Purpose |
PORT | 8054 | HTTP listen port |
DATABASE_URL | postgres://jil:jil@jil-postgres:5432/jil | PostgreSQL connection |
SIGNING_SECRET | devnet-hq-signing-secret | HMAC key for signing image digests + distribute payloads |
SIGNING_KEY_ID | jilhq-signing-key-01 | Key identifier recorded in signed images |
PG_POOL_MAX | 20 | PostgreSQL connection pool size |
5. HMAC Signing Functions
signDigest - Image Signing
function signDigest(digest: string): { signature: string; algorithm: string; signedBy: string }
// HMAC-SHA256 over image digest using SIGNING_SECRET
// Returns base64 signature, used by POST /v1/registry/images/:name/sign
signDistributePayload - Distribution Signing
function signDistributePayload(image, tag, digest, timestamp): string
// HMAC-SHA256 over JSON.stringify({ image, tag, digest, timestamp })
// Sent to validators with each pull command
verifySignature - Image Verification
function verifySignature(digest: string, signature: string): boolean
// Recomputes HMAC-SHA256 and compares against stored signature
// Used by POST /v1/registry/images/:name/verify and verify-deployment
6. Image Registry
JILHQ Registry
| Property | Value |
| Registry | 167.235.150.16:5000 |
| Type | Docker Registry v2 (self-hosted on dedicated JILHQ server) |
| Auth | HMAC-signed pull commands from JILHQ |
| Security | JILHQ pins sha256 digests; validators verify after pull |
Architecture decision: Validators pull from the JILHQ registry (167.235.150.16:5000), a self-hosted Docker Registry v2 on the dedicated JILHQ server. JILHQ pins sha256 digests for every image. After pulling, the validator-update-agent verifies all pulled image digests match JILHQ's pinned manifest. If any digest mismatches (e.g., tampered image), the agent refuses to deploy and reports the mismatch.
7. Distribution Pipeline
Triggered by deploy-to-hetzner.sh --distribute
Step 1: docker build on DevNet (Hetzner portal)
Step 2: docker save | ssh docker load to portal server
Step 3: POST /v1/registry/release → Register + sign + pin digest in JILHQ
Step 4: JILHQ sends refresh command to target validators
→ For each node:
→ Agent pulls from JILHQ registry (167.235.150.16:5000)
→ Agent verifies sha256 digests against JILHQ pinned manifest
→ On match: docker compose up -d (deploy)
→ On mismatch: REFUSE to deploy, report failure
Part II - Validators (Hetzner TestNet)
1. Source Files
The validator-update-agent is a lightweight sidecar running on each Hetzner validator (port 8055). It receives HMAC-signed pull commands from JILHQ, pulls images from the JILHQ registry, restarts containers, and reports back.
| File | Purpose |
services/validator-update-agent/src/index.ts | Express server: HMAC-verified pull+restart, self-registration, Docker API via dockerode |
services/validator-update-agent/package.json | Dependencies: express, pino, pino-pretty, dockerode |
services/validator-update-agent/tsconfig.json | TypeScript config (ES2022, commonjs, strict) |
services/validator-update-agent/Dockerfile | Multi-stage build (node:20-alpine), exposes 8055 |
deploy/hetzner/docker-compose.validator.yml | Added validator-update-agent sidecar with Docker socket mount |
2. REST Endpoints
| Method | Path | Auth | Purpose |
| POST | /v1/pull | HMAC-SHA256 | Pull image from registry, restart container |
| GET | /v1/status | None | List all running containers with digests |
| GET | /health | None | Health check (Docker connectivity) |
POST /v1/pull Flow
- Verify HMAC signature against shared secret
- Check timestamp is within 5-minute replay window
docker pull 167.235.150.16:5000/<image>:<tag>
- Verify pulled digest matches signed digest
- Find running container using that image
- Skip if container is
validator-update-agent (self-restart prevention)
docker stop → docker remove → docker create → docker start
- Report back to JILHQ via
POST /v1/fleet/nodes/:nodeId/verify-deployment
3. Environment Variables
| Variable | Default | Purpose |
PORT | 8055 | HTTP listen port |
NODE_ID | node-primary | This validator's identifier |
JILHQ_URL | https://hq.jilsovereign.com | JILHQ endpoint via Cloudflare (registration + verification) |
JILHQ_SHARED_SECRET | (production secret) | HMAC shared secret (must match JILHQ SIGNING_SECRET) |
REGISTRY_URL | 167.235.150.16:5000 | JILHQ Registry (image source) |
EXTERNAL_IP | 127.0.0.1 | This node's external IP (for self-registration URL) |
AGENT_URL | http://${EXTERNAL_IP}:${PORT} | Override: full agent URL reported to JILHQ |
4. HMAC Verification Functions
verifyHmac - Pull Verification
function verifyHmac(payload: Record<string, unknown>, signature: string): boolean
// HMAC-SHA256 over JSON.stringify({ image, tag, digest, timestamp })
// Uses crypto.timingSafeEqual for constant-time comparison (no timing attacks)
checkTimestamp - Replay Protection
function checkTimestamp(timestamp: string): boolean
// Rejects requests with timestamp > 5 minutes from now
// REPLAY_WINDOW_MS = 300000 (5 min)
5. Docker Service
| Property | Value |
| Image | 167.235.150.16:5000/validator-update-agent:latest |
| Container | validator-update-agent |
| Port | 8055:8055 |
| Volume | /var/run/docker.sock:/var/run/docker.sock |
| Memory | 128M |
| Network | jil-network |
| Restart | unless-stopped |
6. Firewall & Network Config
iptables/ufw Rules (port 8055)
iptables -I INPUT -p tcp --dport 8055 -s 46.225.232.55 -j ACCEPT
ufw allow from 46.225.232.55 to any port 8055 proto tcp
Applied on all 10 Hetzner validators. Only portal IP (46.225.232.55) and JILHQ IP (167.235.150.16) can reach port 8055.
JILHQ Access
// All validators reach JILHQ via Cloudflare:
JILHQ_URL=https://hq.jilsovereign.com
// Cloudflare Worker routes to Hetzner JILHQ origin (167.235.150.16)
Cloudflare routing provides TLS termination and CDN for JILHQ access.
7. Self-Registration Flow
- Agent starts on validator → listens on
:8055
- After 3 seconds, calls
POST ${JILHQ_URL}/v1/fleet/nodes/${NODE_ID}/register-agent
- JILHQ upserts
hq_nodes.metadata.agent_url (auto-creates node if not exists)
- Every 5 minutes, re-registers to keep endpoint fresh
- Distribution reads
metadata->>'agent_url' from hq_nodes - no hardcoded IPs
Part III - Audit & Verification
1. Security Controls Audit
Authentication & Authorization
| Control | Implementation | Location | Status |
| Image signing | HMAC-SHA256 over digest | jilhq signDigest() | Active |
| Distribution payload signing | HMAC-SHA256 over {image, tag, digest, timestamp} | jilhq signDistributePayload() | Active |
| Pull command verification | HMAC-SHA256 with timingSafeEqual | validator-update-agent | Active |
| Replay protection | 5-minute timestamp window | validator-update-agent | Active |
| Shared secret | SIGNING_SECRET / JILHQ_SHARED_SECRET must match | Both services | Required |
| Image must be signed before distribution | Checked in POST /v1/fleet/distribute | jilhq | Enforced |
| Self-restart prevention | Agent skips its own container | validator-update-agent | Active |
Network Access Controls
| Rule | Source | Destination | Port | Protocol |
| Hetzner iptables | Portal IP (46.225.232.55) | Validator | 8055 | TCP |
| Cloudflare routing | All validators | hq.jilsovereign.com | 443 | HTTPS |
Data Integrity
| Check | Where | How |
| Digest verification after pull | validator-update-agent | Compare pulled RepoDigests against signed digest |
| Deployment verification callback | agent → jilhq | POST /v1/fleet/nodes/:nodeId/verify-deployment |
| Signature verification on verify-deployment | jilhq | Recomputes HMAC and compares to stored |
| Audit log on every action | jilhq hq_audit_log | All distribute, sign, register actions logged |
Container Security
| Control | Service | Setting |
| Read-only filesystem | jilhq (HQ compose) | read_only: true |
| No privilege escalation | jilhq (HQ compose) | security_opt: [no-new-privileges:true] |
| Memory limits | All services | jilhq: 256M, agent: 128M |
| Docker socket access | validator-update-agent only | /var/run/docker.sock mount |
2. Code Completeness Checklist
JILHQ
services/jilhq/src/index.ts contains distribute, register-agent, agents endpoints
services/jilhq/migrations/009_distribution_jobs.sql creates hq_distribution_jobs
--distribute flag in deploy-to-hetzner.sh
- SIGNING_SECRET production startup guard
- HMAC auth on read endpoints (image-manifest, digests)
- All 5 distribution endpoints functional
- All 3 signing functions implemented
- hq_distribution_jobs, hq_nodes.metadata, hq_images, hq_image_deployments, hq_audit_log tables active
Validators
services/validator-update-agent/src/index.ts exists and compiles
package.json has express, pino, dockerode
Dockerfile multi-stage build, exposes 8055
- validator-update-agent service in docker-compose.validator.yml
- POST /v1/pull with HMAC verification, digest check, self-restart prevention
- Self-registration on startup (3s delay) + every 5 minutes
- Hetzner iptables rules for port 8055 (portal + JILHQ IPs)
- Mandatory digest verification before deploy (refuses on mismatch)
3. Currently Deployed State
| Component | Location | IP:Port | Status |
| JILHQ Registry | Hetzner | 167.235.150.16:5000 | Active |
| jilhq | Hetzner | hq.jilsovereign.com (167.235.150.16) | Healthy |
| validator-update-agent | jil-mainnet-genesis | 46.225.160.152:8055 | Registered |
| validator-update-agent | jil-validator-us | 5.78.117.94:8055 | Registered |
| validator-update-agent | jil-validator-de | 46.225.116.22:8055 | Registered |
| validator-update-agent | jil-validator-eu | 95.216.146.199:8055 | Registered |
| validator-update-agent | jil-validator-sg | 5.223.74.49:8055 | Registered |
| validator-update-agent | jil-validator-ch | 46.225.87.88:8055 | Registered |
| validator-update-agent | jil-validator-jp | 5.223.49.244:8055 | Registered |
| validator-update-agent | jil-validator-gb | 95.217.14.159:8055 | Registered |
| validator-update-agent | jil-validator-ae | 91.98.193.101:8055 | Registered |
| validator-update-agent | jil-validator-br | 46.224.81.203:8055 | Registered |
4. Key Constants
| Constant | Value | Location | Purpose |
| REPLAY_WINDOW_MS | 300000 (5 min) | validator-update-agent | Timestamp replay protection |
| Distribution timeout | 30000 (30s) | jilhq distribute | Per-node pull timeout |
| Re-registration interval | 300000 (5 min) | validator-update-agent | Self-registration refresh |
| Registration delay | 3000 (3s) | validator-update-agent | Startup delay before first registration |
| Container stop timeout | 10s | validator-update-agent | Graceful stop timeout |
5. Deployment Verification Commands
# 1. JILHQ is healthy (via Cloudflare)
curl -s https://hq.jilsovereign.com/health
# Expected: {"status":"healthy","service":"jilhq",...}
# 2. HMAC auth enforced on read endpoints (no auth = 401)
curl -s https://hq.jilsovereign.com/v1/registry/image-manifest
# Expected: 401 {"error":"HMAC authentication required"}
# 3. Admin auth enforced on write endpoints (no key = 401)
curl -s -X POST https://hq.jilsovereign.com/v1/registry/release -H "Content-Type: application/json" -d '{}'
# Expected: 401 {"error":"Admin API key required"}
# 4. All agents registered (requires admin API key)
curl -s https://hq.jilsovereign.com/v1/fleet/nodes -H "x-api-key: ${HQ_ADMIN_API_KEY}"
# Expected: 10 healthy mainnet validators
# 5. Agent health on each validator
for IP in 46.225.160.152 5.78.117.94 46.225.116.22 95.216.146.199 5.223.74.49 \
46.225.87.88 5.223.49.244 95.217.14.159 91.98.193.101 46.224.81.203; do
curl -s http://$IP:8055/health
done
# Expected: {"status":"ok","docker":"connected"} for each
# 5. End-to-end test
./deploy-to-hetzner.sh --service settlement-api --distribute
# Expected: image built, pushed, registered, signed, distributed