Commit the event
Canonicalise the payload, insert the event and one delivery per subscribed destination, then commit. Only then is the producer acknowledged.
Portfolio case study / Backend infrastructure
A self-hostable event delivery platform built so that once an event is acknowledged, its deliveries exist. PostgreSQL is the queue, workers hold fenced leases, and every retry is a persisted attempt you can inspect.

01 / The problem
Most first implementations are a fetch inside a request handler. It works until a receiver returns 500 during a deploy, a worker crashes between sending and recording the result, or an operator asks whether invoice 1042 reached billing and what exactly was sent.
RelayForge treats those as the normal case. Events and their intended deliveries commit in one transaction before the API returns 202. Workers claim from the deliveries table with FOR UPDATE SKIP LOCKED, take a fresh lease token, sign each request, classify the response, and finalise behind a fence.
02 / What it does
Canonicalise the payload, insert the event and one delivery per subscribed destination, then commit. Only then is the producer acknowledged.
Workers claim due rows with SKIP LOCKED and receive a random fencing token, so a stalled worker cannot overwrite a newer attempt.
HMAC-SHA256 over the timestamp and raw body, a bounded timeout, no redirects, and a response classification that decides retry or terminal failure.
Bounded exponential backoff with jitter, a durable dead-letter queue, and audited replay that creates a new delivery without rewriting history.
The queue is the deliveries table. Enqueue and state transitions share the ingest transaction, so there is no second datastore and no dual-write problem.
03 / Trade-offs
Ingest covers the event and its fan-out in one transaction. The response is sent after commit, which removes the classic accepted-but-not-queued failure mode.
Trade-off: an event with 20 destinations writes 21 rows before responding. That is the correct trade for a delivery system.
A worker id is not enough: the same process can stall past its lease and wake up with a stale result. Every claim takes a fresh token, and every finalise is guarded by that token and state = 'IN_FLIGHT'.
Failure path: a missed guard closes the attempt as ABANDONED; recovery closes expired work as LEASE_EXPIRED.
The Chaos Lab controls a separate receiver process. The worker is unaware of it, so the retries, attempt durations, next retry times and recovery all come from the same code paths as normal traffic.
Consequence: a signing regression shows up as a real 401 in the lab, not as a fake animation.
04 / Chaos Lab
Choose a failure mode, generate traffic, then restore the receiver. The screenshots below are two moments in the same run: every attempt to the controlled destination is returning 500, then the retry queue drains after the receiver is healthy again.
The demo uses an accelerated retry policy (1 second base, 8 second cap, 6 attempts) so the whole arc fits in a minute. The production defaults are 15 seconds, 6 hours and 8 attempts. The acceleration is labelled in the UI.


What is real: the API commit, the queue rows, the worker claims, the signed HTTP requests, the retry schedule, the dead letters and the replay audit. What is simulated: the demo receiver, its failure modes and the generated payloads. Those are labelled in the UI.
05 / Delivery semantics and security
idempotency-key so receivers can deduplicate.(project_id, idempotency_key). Same key and different payload returns 409.replay_of_delivery_id and an audit entry. The original is never mutated.project_id in SQL. Cross-project identifiers return 404, not 403.Not claimed: exactly-once delivery, ordering across deliveries, multi-region replication, per-tenant fairness, production uptime, or a hosted service. The architecture notes state those limits explicitly.
06 / Testing and measurement
Retry-After, timeouts, socket drops, expired leases and stale-worker fencing.| Scenario | Ingest | Ingest p95 | Deliveries | Drain |
|---|---|---|---|---|
| 50 events · concurrency 4 | 552 events/s | 10.2 ms | 150 | 524 ms |
| 500 events · concurrency 8 (repeat) | 638 events/s | 18.6 ms | 1,500 | 2.6 s |
| 2,000 events · concurrency 40 | 739 events/s | 114.1 ms | 6,000 in burst | 13.1 s |
Measured 2026-11-01. All runs report 100% delivery success. The demo receiver answers instantly, so these end-to-end latencies are a lower bound. Full methodology and caveats are in the repository benchmark notes; no number is extrapolated or copied from a different machine.
07 / Limits and next
RelayForge is a portfolio-scale, self-hostable system. It is not a hosted commercial service, has no real customers and makes no uptime claim. The implementation covers the demonstrated scope; the operational edges are documented.
Build disclosure: RelayForge was directed by a human technical lead and implemented with an AI coding agent. The repository records what was human, what was machine, and which claims were verified locally. No public repository or hosted backend demo is published. The interactive preview is a browser simulation; the screenshots, recording, tests and measurements documented here are the evidence for the real backend.
08 / Screenshots
The controlled receiver is a separate process. Only the Demo Receiver destination is affected; Billing and Analytics keep delivering.