Context / need
Client sites in production and internal services generate events that deserve better than a lost log line: contact forms, webhooks, application alerts. A central point was needed to receive, persist and redistribute them, notifications included, reliably, even when a recipient is unavailable or the service restarts.
What makes this piece distinctive: the same person writes the API and operates it in production. Every development choice gets confronted there with its operational consequences.
Constraints
- Guaranteed delivery. An accepted event must never be lost, whether the recipient is down or the service is being redeployed.
- Publicly exposed ingest, since sites must be able to push from the Internet: authentication and rate limiting are mandatory.
- Private administration. Reading and management only exist on the zero-trust network.
Options considered
- A dedicated broker (RabbitMQ, Redis with a queue): delivery reliability is delegated, but it is one more critical stateful piece to operate, for a throughput that does not justify it.
- Fire-and-forget with logs: unacceptable. Silent loss is exactly the problem to solve.
- PostgreSQL as a durable queue: the database is already there, backed up, well understood. The delivery queue becomes a table, and the guarantees come from transactions.
Decision & why
A NestJS + Prisma + PostgreSQL API, with the retry queue in the database. Every failed delivery is retried with exponential backoff, and the queue state survives restarts because it is persisted. The detailed design of that queue has its own write-up: Designing a persisted retry queue.
The deployment is single-host with two faces. The same service exposes ingest publicly (rate-limited, authenticated), while the administration routes only respond via the tailnet. The boundary is declared at the edge (reverse proxy), without relying on the code alone.
On the operations side, the API is treated as a full production service:
- API tokens stored hashed (SHA-256), never a plaintext credential in the database;
- rate limiting on the public ingest;
- structured logs, usable rather than decorative;
- generated OpenAPI: the contract is documented and testable;
- graceful shutdown: a redeployment cleanly finishes what is in flight, without cutting deliveries halfway.
Accepted tradeoff
PostgreSQL as a queue does not compete with a real broker, neither in throughput nor in advanced semantics (complex fan-out, consumer groups). At the actual scale of the need, it is the right compromise: one less critical piece, transactional guarantees already understood, and a queue readable in SQL when diagnosis is needed. If throughput changes by an order of magnitude, the publishing interface allows swapping the implementation without touching the rest.
Outcome
In production since May 2026: four sites push their events to it. The volume is modest today (forms and transactional emails from the delivered sites); the machinery is sized for what comes next, including server monitoring and operations events.
- Reliable: no silent loss. What is accepted is persisted, what fails is retried, what restarts resumes where it left off.
- Maintainable: modular NestJS, typed Prisma, an OpenAPI contract. The queue reads in SQL.
- Secure: ingest authenticated with hashed tokens and rate-limited, administration nonexistent from the Internet.
What ops changed in the dev
Operating your own code changes how you write it. Three concrete examples on the Hub:
- Graceful shutdown matters. A deployment must go unnoticed, without cutting deliveries halfway and sending them into retry. The service drains before shutting down.
- Logs are structured because they get read. When you are also the on-call, an unusable log costs you at 11 pm. Every event carries what it takes to be traced from ingest to delivery.
- Rate limiting also protects the database. The public endpoint is sized to absorb abuse without degrading the rest of the platform.