Daniel Isaac
← All writing

Writing

Exactly-once money movement: surviving retries and duplicate webhooks

Why check-then-act double-charges under concurrency, and how claiming a unique key at the database turns at-least-once delivery into exactly-once processing.

Published
2026-08-01
Topics
Reliability · Payments · Concurrency

Most money bugs are not exotic. They come from two boring facts:

  1. Operations fail halfway, so something retries them.
  2. Payment providers deliver callbacks at least once, which means sometimes more than once, sometimes out of order, sometimes minutes late.

Put those together and the naive version of a payment handler charges a customer twice, or credits a wallet twice, and you usually find out at reconciliation instead of in code review. The whole job is turning at-least-once delivery into exactly-once processing.

The fix that looks right and isn’t

The instinct is to check before you act:

if (!alreadyPaid(orderId)) {
  charge(orderId)
  markPaid(orderId)
}

This is broken the moment there is more than one worker, one server, or one redelivered webhook, which in production there always is. Two callbacks for the same order arrive at the same time. Both run alreadyPaid, both read “no,” both call charge. The gap between the read and the write is where the double charge lives. In-memory locks do not save you either, because they do not span processes or instances.

Push the decision to where the race is actually settled

The database is the one place all your workers agree on truth. So make it the referee. Claim a unique key for the operation before you perform the side effect, and let a unique constraint reject the duplicate.

create table idempotency_keys (
  key         text primary key,      -- e.g. the provider reference
  status      text not null,         -- PENDING | DONE
  response    jsonb,
  created_at  timestamptz default now()
);

// returns false if the key already exists (duplicate); the unique
// constraint is what makes this safe under concurrency
const claimed = insertIfAbsent(key)
if (!claimed) return storedResponseFor(key)   // replay, do not re-charge

const result = charge(...)
finalize(key, result)                          // status = DONE, store response
return result

Two callbacks racing on the same key now collide on the primary key. Exactly one insert wins and does the work; the other reads the stored result and returns it. There is no window between the check and the act, because the check is the act.

For state transitions the same idea takes an even simpler shape: make the update itself conditional and atomic.

update payments
   set status = 'PAID'
 where id = $1
   and status in ('PENDING', 'PROCESSING');   -- only a real transition flips a row

If the affected-row count is zero, someone already moved it. You return without crediting anything. One statement, no lock, no gap.

What this looks like in a real project

My Job Hunt CRM (open source) ingests webhooks this way. Every inbound event carries a provider reference; the ingestion path claims that reference before it applies the event, so a redelivered webhook maps to the same claim and is applied exactly once. The event is then written to an append-only log rather than mutating a row in place, which means a duplicate is not just prevented, it is visible: you can see the second delivery arrive and no-op.

The parts people skip

  • Choose the key deliberately. It has to be stable across retries and unique per logical operation. A provider’s transaction reference is usually right; a timestamp or a random id is not.
  • Store the response, not just the fact. A duplicate should get the same answer the first call got, so store the result under the key and replay it.
  • Know the delivery contract. Providers promise at-least-once, never exactly-once. Exactly-once is something you engineer on your side; do not assume the network gives it to you.
  • Keep reconciliation as the backstop. Idempotency prevents double application. A scheduled reconciliation against the provider catches the cases where you never heard back at all. You want both.

Why it matters

The difference between “it usually works” and a system you can put real money through is almost entirely in these paths, the ones that only fire under load, retries, and duplicate callbacks. They do not show up in a happy-path demo. They show up in production, and by then a wrong answer costs a refund and an apology, or a chase for money you already collected. Designing for exactly-once from the start is cheaper than reconciling your way out of it later.

← All writing