Designing Idempotent and Resilient APIs

The network will drop your response after the work is done but before the client hears back. If your API can't tell a retry from a brand-new request, that's how a customer gets charged twice.

Designing Idempotent and Resilient APIs - diagram by Sushan Khadka (namelessnerd)

What idempotency actually means

An operation is idempotent if doing it twice has the same effect as doing it once. Flipping a light switch to "on" is idempotent — the light is on whether you flip it once or five times. Incrementing a counter is not — each call changes the result.

HTTP bakes this in, and it is worth being precise because people misremember it. GET, PUT, and DELETE are defined as idempotent; POST and PATCH are not. PUT /users/42 {name: "Sam"} sets the record to a known state — run it ten times, same result. POST /users {name: "Sam"} is meant to create, so ten calls arguably means ten users. That gap between "safe to repeat" and "not safe to repeat" is exactly where money gets double-charged, so the interesting work is making the non-idempotent operations behave idempotently.

Retries are not optional

Timeouts, load balancer hiccups, mobile dead zones — clients will retry, and they should. A client that gives up on the first network blip is a fragile client. The classic failure: the server charges the card, the response is lost in transit, the client's timeout fires, it retries, and now there are two charges. The retry isn't the bug. The bug is an endpoint that treats the second POST as new work.

This is the crucial mental shift: in a distributed system, the absence of a response tells you nothing. The request might have failed before doing anything, failed after doing everything, or succeeded with only the reply lost. The client cannot distinguish these, so it must retry — and the server must be ready to receive that retry safely.

Idempotency keys + a dedup store

The fix is boring and bulletproof. The client sends a unique idempotency key — a UUID it generates once per logical operation — in a header with each request. The server records that key alongside the result in a dedup store, and replays the stored response if the key shows up again.

POST /charges HTTP/1.1
Idempotency-Key: 3f9a1c7e-2b04-4c1a-9e77-8b1d2f0a5c33
Content-Type: application/json
 
{ "amount": 4200, "currency": "usd", "source": "card_abc" }

The server logic:

First request: do the work, persist (key, result), respond — ideally in the same database transaction so the charge and the key commit together or not at all.
Any repeat with the same key: skip the work, return the saved result byte-for-byte.
Keys live in the request, never derived server-side — only the client knows two calls are "the same" logical operation.

This is exactly how Stripe, PayPal, and every serious payments API handle it. The key is the client's promise: "if you see this twice, it is my retry, not a new intent."

Exactly-once is a myth (the effect isn't)

You cannot guarantee a message is delivered exactly once — that's a distributed-systems impossibility, provable from the fact that the sender can never be sure its acknowledgment arrived. What you can guarantee is exactly-once effect: the operation lands once no matter how many times it's delivered.

The practical recipe is at-least-once delivery plus idempotent processing equals effectively-once. Let the network deliver a message one or more times — that part is cheap and reliable. Make the handler idempotent so duplicates are harmless — that part is your job. Together they give you the outcome everyone actually wanted. Stop chasing the delivery myth; design for the effect.

The concurrent-retry race

Here is the subtle bug that survives a naive implementation. Two retries can arrive at the same millisecond, both check the dedup store, both find no existing key, and both proceed to charge the card. An if-exists lookup won't save you — there's a gap between the read and the write where both requests are in flight, each believing it is the first.

Push correctness into the database with a unique constraint and let one writer win:

CREATE UNIQUE INDEX idx_idempotency ON requests (idempotency_key);
-- the second concurrent INSERT fails with a violation;
-- catch it, then return the first request's stored result

The loser catches the constraint violation and returns the winner's response instead of erroring. The race is settled by the one component that's actually serializable — your storage engine. Application-level "check then act" is always racy under concurrency; a unique index is atomic by definition.

Resilience is more than idempotency

Idempotency makes retries safe; a few companion patterns make the whole system stable under failure:

Exponential backoff with jitter. Retrying immediately, in lockstep, turns a blip into a self-inflicted DDoS as every client hammers back at the same instant. Back off geometrically and add randomness so the herd disperses.
Circuit breakers. When a downstream dependency is clearly down, stop calling it for a cooldown and fail fast, rather than piling up requests that will all time out anyway.
Timeouts and bulkheads. Every remote call needs a timeout, and isolating pools of resources per dependency keeps one slow service from consuming every thread and taking the whole app down.
Set a key expiry. Idempotency keys don't need to live forever — expiring them after 24–48 hours keeps the dedup store from growing without bound, comfortably longer than any sane client will retry.

The takeaway

Know which verbs are idempotent — GET/PUT/DELETE are; make POST/PATCH behave that way deliberately.
Assume retries, then make them free: same key in, same result out.
Chase the effect, not the delivery — at-least-once delivery plus idempotent handling equals effectively-once.
Let the database referee concurrent retries with a unique constraint, not application-level checks.
Back off with jitter and trip circuit breakers — safe retries still need to be paced retries.