Webhooks that survive load-shedding

A payment webhook is the gateway telling your server that money moved. If your server is down when it fires, because of load-shedding, a network drop or a deploy, you must not lose that order. The patterns that make webhooks durable are the same everywhere, and they matter more in South Africa because outages are routine, not rare. This guide is the reference implementation in four rules, with the code.

Why is the redirect not enough?

Every gateway offers two signals that a payment happened: the browser redirect back to your success page, and the webhook to your server. Teams get burned by treating the first one as the truth.

The redirect is the customer’s browser, and the customer’s browser is not a reliable witness. It never arrives if their battery dies between paying and redirecting, if the mobile network flaps, or if they just close the tab. It can also be forged: a success URL that marks orders paid is an invitation to type that URL by hand. The webhook is the gateway’s own server talking to yours, it can be cryptographically verified, and the gateway retries it until your server confirms. The redirect is UX; the webhook is the record.

How do you make a payment webhook durable?

Four rules, none optional:

• Verify the signature. Every serious gateway signs its webhooks: Paystack signs the payload with your secret key, Stitch sends an HMAC-SHA256 signature header, iKhokha signs payloads with HMAC-SHA256, PayFast validates its ITN posts with a passphrase and source checks. Check the signature server-side and reject anything that fails, so a forged request can never mark an order paid.
• Confirm server-side, never on the browser redirect. The success page says “thank you, we are confirming your payment” and nothing else. The order flips to paid only when the verified webhook lands.
• Be idempotent. Gateways retry, and retries mean the same event will arrive more than once. Key on the gateway’s event or transaction id, and make processing it twice a no-op. A duplicate webhook that double-credits an order or sends two shipments is a self-inflicted bug.
• Accept fast, work off a queue. Return 200 the moment the event is verified and stored. Send the confirmation email, call the courier API and update the ledger from a worker afterwards. A handler that does slow work inline will time out exactly when things are already going wrong, and a timeout reads as failure to the gateway.

The reason these four hold up under load-shedding is the interplay between the gateway’s retries and your queue: the retries cover the window your server was unreachable, and the queue covers everything after the event is safely yours.

What does an outage actually look like?

The scenario this guide is named for. Your server is in your office; stage 4 takes the suburb down in the evening rush:

The order survived a two-and-a-half-hour outage with no human involved. That is the whole point: durability here is not heroics, it is refusing to lose a race you were never required to win in real time.

What should the handler actually do?

The shape, in TypeScript. Adapt the header name and signature scheme to your gateway’s docs:

app.post('/webhooks/payments', async (req, res) => {
  // 1. Verify: reject anything the gateway did not sign.
  const expected = crypto
    .createHmac('sha256', process.env.WEBHOOK_SECRET)
    .update(req.rawBody)
    .digest('hex');
  if (!timingSafeEqual(expected, req.headers['x-signature'])) {
    return res.status(401).end();
  }

  const event = JSON.parse(req.rawBody);

  // 2. Idempotency: the event id is unique, inserting it twice fails.
  const fresh = await db.insertIgnoreConflict('webhook_events', {
    id: event.id,
    payload: req.rawBody,
  });
  if (!fresh) return res.status(200).end(); // seen before: ack and stop

  // 3. Queue the slow work; ack immediately.
  await queue.enqueue('process-payment-event', { eventId: event.id });
  return res.status(200).end();
});

Three details carry the weight. The signature check uses a constant-time comparison over the raw body, not the parsed JSON, because re-serialised JSON is not byte-identical. The insert with a unique key is the idempotency test and the storage in one step, so there is no gap where a duplicate can slip between check and write. And the handler acknowledges before the real work happens, so a slow courier API cannot make the gateway think delivery failed.

The worker that processes the queue should itself be idempotent, checking the order’s current state before acting, because queues also redeliver. The rule generalises: assume every message in the system can arrive twice.

How do you cover the gap the retries don’t?

Gateway retries are generous but finite, and their schedules differ by provider. Two habits close the remaining window:

• Reconcile on a timer. A job every 15 to 30 minutes asks the gateway’s API about any order still pending after an hour. If the gateway says paid and your database says pending, you found a lost webhook and fixed it without a support ticket. This also covers the failure the retry model cannot: a bug in your own handler that returned 200 and then crashed.
• Rehearse the outage in sandbox. PayFast’s sandbox fires test ITNs, Paystack’s test mode sends real signed events, Stitch’s sandbox does the same. Point them at your handler, kill the process mid-delivery, bring it back, and watch the retry land. Ten minutes of rehearsal beats discovering the recovery path during an actual stage 6 evening.

Do the four rules, the reconciliation job and one rehearsal, and load-shedding stops being a payments risk at all: the power still goes out, but the orders stop caring. Then take the fee side of the same rigour to how to verify a gateway’s real fees.