The problem
A central service receives events and must deliver them: notifications, outgoing webhooks. Recipients go down, restart, time out. Two classic failures:
- Silent loss. The send fails, you log it, you move on. The message no longer exists, and nobody notices until it matters.
- In-memory retry. Better, until the first redeployment, when the queue disappears with the process.
The pattern
Treat delivery as persisted state:
- Each delivery is a row in PostgreSQL: event, destination, attempt count, due time of the next attempt, status.
- Exponential backoff. Each failure pushes the next due time further out. The recipient gets time to recover instead of being hammered.
- A worker claims what is due, meaning the deliveries whose due time has passed, transactionally: the state never lies.
- Recovery for free. After a crash or a deployment, the queue is still there. The worker resumes exactly where the state says. Combined with graceful shutdown, a redeployment cuts nothing halfway.
- Bounds and visibility. Past a number of attempts, the delivery moves to permanent failure: visible, queryable in SQL, never evaporated.
Why not a broker?
RabbitMQ or an equivalent provides all of this natively, at the cost of one more critical stateful piece to operate, back up and monitor. On a single-node platform whose database is already backed up and well understood, the compromise leans toward PostgreSQL: the guarantees come from transactions, and the diagnostic tooling is SQL. The day throughput demands it, the publishing interface allows changing the implementation without touching the rest.
Lessons
- Idempotency is half the pattern. A retry can deliver twice; the recipient must be able to absorb it.
- Backoff protects both ends. The recipient gets room to breathe, and the sender does not burn its resources on an unreachable service.
- A permanent failure must be visible. A queue that quietly swallows errors reproduces the original problem, with one extra step.