Backend reliability
Ledgerloop: money movement that stays correct under failure
An open-source playground that makes three production reliability patterns runnable and visible: exactly-once webhook processing, a guarded state machine over an append-only event log, and a reconciler that catches drift. The runnable companion to my reliability essays.
- Role
- Author, open source
- Period
- 2026
- Stack
- Next.js · TypeScript · better-sqlite3 · node:test
- Source
- GitHub ↗
Ledgerloop takes the three reliability patterns I lean on in payments work and makes them runnable in a browser: exactly-once webhook processing, a guarded state machine over an append-only event log, and a reconciler that catches drift. It is the runnable companion to my two reliability essays, and it is open source.
Ledgerloop is a small, deliberate demonstration, not a product. Its job is to make three patterns you usually only read about visible and testable. Send a duplicate webhook and watch it get swallowed. Try an illegal state transition and watch the guard reject it with its own reasoning. Corrupt the cache and watch reconciliation catch the drift. Anything you can claim about reliability, you can click here and verify.
The problem
Reliability patterns are easy to describe and hard to believe until you have seen them fail correctly. Most writing about idempotency or state machines stops at the diagram. I wanted the opposite: a thing you can open, poke, and watch hold under the exact conditions that break naive code, concurrent duplicates, illegal moves, and a cache that quietly drifts from the truth.
Architecture
One payment moves through a lifecycle. Inbound webhooks clear an idempotency ledger before they touch anything, so a redelivered event maps to the same claim and runs once. Every accepted transition is guarded by a legal-moves table and, in a single transaction, appends an immutable event and updates a fast status cache. The event log is the source of truth. The cache is a convenience the reconciler can rebuild and check at any moment.
Decisions and tradeoffs
Claim the idempotency key before doing any work
A handler that checks “already processed?” and then acts has a gap between the check and the write. Under two concurrent deliveries, both checks pass and the work runs twice. Ledgerloop claims the key first, behind a unique constraint, so the check is the act. A duplicate collides on the claim and returns the stored outcome instead of moving money again.
Rejected: check-then-act. It races the moment there is more than one worker, and in production there always is.
One status and a table of legal moves, guarded
Status is a single value with an explicit adjacency table of legal transitions. Every change goes through one guard, so an illegal move throws at the boundary and comes back as a 409 that quotes the machine’s own reasoning, rather than silently writing something impossible. The transition and its event are committed in the same transaction, so the log can never disagree with the row.
Rejected: boolean flags (is_paid, is_failed, is_refunded). Four of them describe sixteen states, most of them nonsense, and something always forgets which combinations are real.
The cache is disposable, the log is the truth
State lives in two places: the append-only event log and a cached current status. The cache exists only so reading the latest status is fast. The reconciler folds the log from the start and compares the result to the cache, so drift is always detectable and never silent. Ledgerloop even lets you inject drift on purpose to watch it get caught. When the two disagree, the log wins.
Rejected: trusting the cache. A cache is only safe if divergence from the source of truth can be detected. Otherwise it quietly becomes the wrong answer.
What it is, honestly
This is a teaching artifact, not a business, and the patterns in it are not novel. They are the ones a careful engineer reaches for, and that is the point. Reliability is easier to trust when you can run it. Ledgerloop turns three claims I make about my work into something a reader can verify in a minute, and the core lib is written to be lifted straight into a real service.
The through-line
These are the same patterns I run in production on the Vida recovery engine, where the thing that must not break is someone’s repayment. Ledgerloop is the open, runnable version, and the companion to the essays where I wrote the patterns down: exactly-once money movement and lifecycles as state machines.