Technical Architecture
Parallel Settlement Pipeline
Fan-out/collect attestation architecture via RedPanda. Gate-based verdicts. Target: 1M settlements/min.
1M/min
Target throughput
~16,667/s
Settlements per second
32
Partitions per topic
<5s
Aggregation timeout
6
Parallel attestation services
~83K
Max concurrent in-flight
This is NOT BFT consensus. Block validators use Byzantine Fault Tolerance - 70% of validators must agree (14-of-20 quorum). Settlement attestation is a gate, not a vote. Every required check must pass. A single "denied" from any service kills the settlement immediately. There is no threshold, no majority, no quorum. The model is fail-closed: if a service is unreachable, the verdict is "Review" (hold), never "Yes".
Gate Logic: ANY check returns "denied" = verdict No / ANY check returns "hold" = verdict Review / ALL checks return "approved" = verdict Yes
Architecture
How It Works
- Request arrives - Settlement request enters via HTTP POST to
/api/v1/settleor via RedPanda topic jil.attest.requests - Fan-out - Aggregator publishes the request to N attestation topics simultaneously. Each service gets the same settlement_id (correlation ID) and processes independently.
- Parallel processing - Each attestation service (sanctions, risk, credentials, policy, ownership, bank) evaluates the settlement against its domain. No service waits for another.
- Result collection - Each service publishes its result to jil.attest.results with the correlation ID. The aggregator collects results in an in-memory map.
- Verdict resolution - When all required checks complete (or 5s timeout fires), the aggregator computes the verdict using gate logic and publishes to jil.settlement.verdicts
- Execution - Settlement consumer reads the verdict. Yes = execute. No = reject with reason codes. Review = hold for manual approval.
Gate Model vs. BFT Consensus
| Dimension | BFT Validators (Block Consensus) | Settlement Pipeline (Gate) |
|---|---|---|
| Model | Vote - 70% quorum (14-of-20) | Gate - ALL must pass |
| Single failure | Tolerated (up to 6 of 20) | Kills the settlement (denied = No) |
| Timeout behavior | Quorum still achievable | Hold verdict (fail-closed) |
| Participants | 20 geographically distributed validators | 4-6 attestation services (same datacenter) |
| Latency tolerance | Seconds (cross-continent) | Milliseconds (same Docker network) |
| Independence | Each validator runs full node | Each service has a single domain |
| Adversary model | Byzantine (malicious nodes) | Operational (service failure/timeout) |
| Purpose | Agree on block state | Verify settlement conditions |
Topic Architecture
| Topic | Direction | Producer | Consumer | Partitions |
|---|---|---|---|---|
| jil.attest.requests | Inbound | Clients / Wallet API | Aggregator | 32 |
| jil.attest.sanctions | Fan-out | Aggregator | Sanctions Screening Cache | 32 |
| jil.attest.risk | Fan-out | Aggregator | Risk Scoring Attest | 32 |
| jil.attest.credentials | Fan-out | Aggregator | Credential Registry | 32 |
| jil.attest.policy | Fan-out | Aggregator | Policy Decision API | 32 |
| jil.attest.ownership | Fan-out | Aggregator | Ownership Verification | 32 |
| jil.attest.bank | Fan-out | Aggregator | Bank Attestation Ingestion | 32 |
| jil.attest.results | Collect | All attestation services | Aggregator | 32 |
| jil.settlement.verdicts | Output | Aggregator | Settlement Consumer | 32 |
| jil.attest.dlq | Dead letter | Aggregator | Ops monitoring | 8 |
Throughput Engineering
| Parameter | Value | Rationale |
|---|---|---|
| Target throughput | 1,000,000/min | 16,667 settlements/second sustained |
| Partitions per topic | 32 | Each partition handles ~520 msgs/sec |
| Fan-out multiplier | 4-6x | Each settlement produces 4-6 fan-out messages |
| Total RedPanda throughput | ~100K msgs/sec | 16.7K requests x 6 fan-outs = ~100K messages/sec across all topics |
| RedPanda capacity | Millions/sec | Well within RedPanda single-node limits |
| Aggregation timeout | 5,000ms | Max concurrent: 16,667/sec x 5s = ~83K in-flight |
| Memory per settlement | ~1KB | Request + partial results = ~83MB total RAM |
| DB write rate | 16,667 verdicts/sec | Async batch insert, connection pool: 50 |
Horizontal Scaling
The pipeline scales horizontally at every layer:
- Aggregator instances: Multiple aggregator instances share the same consumer group. RedPanda assigns partitions across instances. 2 instances = ~32K/sec capacity. 4 instances = ~64K/sec.
- Attestation services: Each attestation service can run N replicas in the same consumer group. Partitions distribute across replicas automatically.
- Database: Verdict writes are async and batched. Connection pool scales to 50 concurrent connections per instance.
- RedPanda: Additional brokers can be added for replication and throughput. 32 partitions support up to 32 consumers per topic.
Failure Modes
| Failure | Behavior | Recovery |
|---|---|---|
| Attestation service down | Timeout fires - verdict = Review | Manual review queue, service restart |
| Attestation service slow | Timeout fires for missing check only | Partial results + hold verdict |
| RedPanda partition offline | Consumer rebalance to healthy partitions | RedPanda self-heals, messages replayed |
| Aggregator crash | In-flight settlements lost (in-memory) | Clients re-submit, Kafka offsets preserve position |
| Database unreachable | Verdicts still published to Kafka topic | Backfill from Kafka when DB recovers |
| Duplicate result | Map.set() overwrites - idempotent | No impact - last result wins |
API Reference
| Method | Endpoint | Description |
|---|---|---|
POST | /api/v1/settle | Submit settlement for attestation. Returns 202 with settlement_id. |
GET | /api/v1/settle/:id | Poll verdict status. Returns in-flight state or final verdict. |
GET | /api/v1/verdicts | List recent verdicts. Filterable by verdict type. Paginated. |
GET | /api/v1/pipeline | Pipeline status, throughput metrics, topic configuration. |
GET | /health | Health check with DB + pending aggregation count. |
GET | /metrics | Prometheus metrics (requests, verdicts, latency, errors). |
Service: settlement-aggregator
Dependencies: RedPanda, PostgreSQL, attestation services
Source: services/settlement-aggregator/