Portfolio case study / Backend infrastructure

RelayForge

A self-hostable event delivery platform built so that once an event is acknowledged, its deliveries exist. PostgreSQL is the queue, workers hold fenced leases, and every retry is a persisted attempt you can inspect.

Independent buildSelf-hostable · Browser previewTypeScript · Fastify · PostgreSQL
  • PostgreSQL
  • Fastify
  • TypeScript
  • SSE
  • Playwright
RelayForge Control Room with measured success rate, throughput, p95 latency, delivery pipeline, live event stream and system health
Real capture Control Room · every KPI is a SQL aggregate over persisted rows
56tests, including 28 against real PostgreSQL
6Playwright E2E flows passed; 4 duplicate guards skipped
739events/s peak ingest in the local benchmark
100%delivery success in the measured runs

01 / The problem

A webhook is easy until the receiver has a bad day.

Most first implementations are a fetch inside a request handler. It works until a receiver returns 500 during a deploy, a worker crashes between sending and recording the result, or an operator asks whether invoice 1042 reached billing and what exactly was sent.

RelayForge treats those as the normal case. Events and their intended deliveries commit in one transaction before the API returns 202. Workers claim from the deliveries table with FOR UPDATE SKIP LOCKED, take a fresh lease token, sign each request, classify the response, and finalise behind a fence.

02 / What it does

Four processes, one database, no broker.

01

Commit the event

Canonicalise the payload, insert the event and one delivery per subscribed destination, then commit. Only then is the producer acknowledged.

02

Claim with a lease

Workers claim due rows with SKIP LOCKED and receive a random fencing token, so a stalled worker cannot overwrite a newer attempt.

03

Sign and deliver

HMAC-SHA256 over the timestamp and raw body, a bounded timeout, no redirects, and a response classification that decides retry or terminal failure.

04

Recover or replay

Bounded exponential backoff with jitter, a durable dead-letter queue, and audited replay that creates a new delivery without rewriting history.

The queue is the deliveries table. Enqueue and state transitions share the ingest transaction, so there is no second datastore and no dual-write problem.

03 / Trade-offs

Design decisions

Acknowledged means durable.

Ingest covers the event and its fan-out in one transaction. The response is sent after commit, which removes the classic accepted-but-not-queued failure mode.

Trade-off: an event with 20 destinations writes 21 rows before responding. That is the correct trade for a delivery system.

Leases are fenced, not named.

A worker id is not enough: the same process can stall past its lease and wake up with a stale result. Every claim takes a fresh token, and every finalise is guarded by that token and state = 'IN_FLIGHT'.

Failure path: a missed guard closes the attempt as ABANDONED; recovery closes expired work as LEASE_EXPIRED.

Failure injection uses the real code path.

The Chaos Lab controls a separate receiver process. The worker is unaware of it, so the retries, attempt durations, next retry times and recovery all come from the same code paths as normal traffic.

Consequence: a signing regression shows up as a real 401 in the lab, not as a fake animation.

04 / Chaos Lab

Break a receiver, then restore it.

Choose a failure mode, generate traffic, then restore the receiver. The screenshots below are two moments in the same run: every attempt to the controlled destination is returning 500, then the retry queue drains after the receiver is healthy again.

The demo uses an accelerated retry policy (1 second base, 8 second cap, 6 attempts) so the whole arc fits in a minute. The production defaults are 15 seconds, 6 hours and 8 attempts. The acceleration is labelled in the UI.

Chaos Lab with the receiver forced to return HTTP 500 and deliveries retrying
Forced failure 500 responses, persisted attempts, scheduled retries
Chaos Lab after restoring the receiver with the retry queue drained and deliveries recovered
Recovered Restore the receiver; the next retry succeeds

What is real: the API commit, the queue rows, the worker claims, the signed HTTP requests, the retry schedule, the dead letters and the replay audit. What is simulated: the demo receiver, its failure modes and the generated payloads. Those are labelled in the UI.

Screen recording Real failure → retry → recovery · Silent · Captured from the running stack

05 / Delivery semantics and security

At-least-once delivery.

Semantics

  • At-least-once delivery. A crash between the HTTP request and the database commit can duplicate an attempt.
  • The delivery id is stable across retries and replays, and is sent as idempotency-key so receivers can deduplicate.
  • Idempotency is scoped to (project_id, idempotency_key). Same key and different payload returns 409.
  • Replay creates a new delivery with replay_of_delivery_id and an audit entry. The original is never mutated.

Security boundaries

  • Project-scoped API keys are stored as SHA-256 hashes; the demo browser session is limited to the demo tenant.
  • Every read and write filters by project_id in SQL. Cross-project identifiers return 404, not 403.
  • Destination URLs are validated on write and on every attempt. Private ranges are blocked; the validated IP is pinned into the TCP connect.
  • Signing secrets are returned only on create or rotation; reads expose a masked prefix and logs redact the value.

Not claimed: exactly-once delivery, ordering across deliveries, multi-region replication, per-tenant fairness, production uptime, or a hosted service. The architecture notes state those limits explicitly.

06 / Testing and measurement

Tests and measurements

Verification

  • 56 tests: 28 PostgreSQL integration, 17 shared-logic, 6 worker and 5 receiver tests.
  • Concurrent exclusive claims, real 500 → retry → dead letter, Retry-After, timeouts, socket drops, expired leases and stale-worker fencing.
  • Idempotency conflicts, payload limits, tenant isolation, SSRF rejection, metrics, health, replay provenance and audit.
  • 6 Playwright E2E flows cover the Chaos Lab break/recover path, trace and replay, destinations metadata and mobile overflow. Four duplicate guards are skipped by design.

What the numbers mean

  • Environment: Ryzen 5 5600, 32 GB, Windows 11, Node 26, PostgreSQL 18.6 on the same machine.
  • One worker process with concurrency 8, three destinations on loopback, and the real HTTP ingest path.
  • Delivery p95 is acceptance to successful delivery, so it includes queue wait and backoff. Per-attempt transport time is 1 to 15 ms.
  • No retries fired in these runs, so the benchmark does not measure the retry path. Integration tests and the Chaos Lab do.
RelayForge local benchmark results
ScenarioIngestIngest p95DeliveriesDrain
50 events · concurrency 4552 events/s10.2 ms150524 ms
500 events · concurrency 8 (repeat)638 events/s18.6 ms1,5002.6 s
2,000 events · concurrency 40739 events/s114.1 ms6,000 in burst13.1 s

Measured 2026-11-01. All runs report 100% delivery success. The demo receiver answers instantly, so these end-to-end latencies are a lower bound. Full methodology and caveats are in the repository benchmark notes; no number is extrapolated or copied from a different machine.

07 / Limits and next

What this build does not claim.

RelayForge is a portfolio-scale, self-hostable system. It is not a hosted commercial service, has no real customers and makes no uptime claim. The implementation covers the demonstrated scope; the operational edges are documented.

  • Known limits: in-process rate limiting, no per-tenant fairness, no retention or partitioning, no durable SSE replay, and metrics computed on read.
  • Not verified here: Docker Compose was authored but not executed, and the GitHub Actions workflow was not run on GitHub. The embedded-PostgreSQL path is what every test and screenshot used.

Build disclosure: RelayForge was directed by a human technical lead and implemented with an AI coding agent. The repository records what was human, what was machine, and which claims were verified locally. No public repository or hosted backend demo is published. The interactive preview is a browser simulation; the screenshots, recording, tests and measurements documented here are the evidence for the real backend.