A drop-in idempotency layer for FastAPI POST endpoints with the proof run in its tests: 50 concurrent duplicate payment requests, one charge in the ledger, 49 byte-identical replays, and a war-story test for the crash window that middleware-only idempotency pretends not to have.
-informational){kind=icon}
- Retries are how networks work and how customers get double-charged; with the shield, sending the same
Idempotency-Keytwice, concurrently or hours apart, returns the same 201 with the same charge id (50-way concurrent storm asserted in tests, one charge per storm across 10 storms in the benchmark). - Concurrent duplicates usually get a 409 and a client-side retry loop nobody writes correctly; here losers wait for the winner and replay its exact response with
Idempotent-Replay: true(ADR-001). - Middleware-only idempotency has a crash window (die after the side effect commits, before the response stores) that re-executes handlers on retry; this repo names the window, tests it, and closes it with a database uniqueness constraint where atomicity actually exists (ADR-002).
Every payment API document says "retry with the same idempotency key", and every backend team eventually learns what that costs to implement correctly: not the happy path (a Redis GET is easy) but the two hard cases. First, the storm: a mobile client on a bad connection fires the same request five times in 200 ms, all before the first one finishes; a naive check-then-set races and charges twice. Second, the crash: the server commits the charge and dies before recording the response; the client retries, the handler runs again, and the ledger has two rows for one tap.
The shield handles both, honestly. Claiming is atomic (SET NX with an in-flight TTL and a per-claimant token); completion is a Lua compare-and-set so only the claimant can store the result; concurrent losers poll until the winner's response exists and replay it byte-identically. For the crash window, the design refuses to lie: no middleware can make a Postgres commit atomic with a Redis write, so the demo payment handler carries a UNIQUE(idem_key) constraint and treats the violation as "already done", resolving to the existing charge. The test suite simulates the exact crash (record wiped after commit, retry re-executes the handler) and asserts the ledger still holds one charge. Both layers are load-bearing, and the ADR says which layer provides which guarantee.
| Technology | Role in this project | Why chosen here |
|---|---|---|
Redis (SET NX + Lua CAS) | Claim, wait, replay store | Atomic claim across replicas; token-guarded completion so a stale claimant cannot overwrite a winner |
| PostgreSQL | The demo ledger and the second guarantee layer | UNIQUE(idem_key) is the only place at-most-once is actually enforceable for a transactional side effect |
| FastAPI decorator | Integration surface | @idempotent(store, settings) on a handler; no framework fork, no global middleware magic |
| httpx + asyncio | Storm benchmark | 50-way concurrent duplicates against the real ASGI app, results asserted not eyeballed |
| structlog | Logs | Completion rejections (stale claimants) are visible, not silent |
| pytest | Suite | 6 tests: storm collapse, distinct keys, sequential replay, missing key, crash window, dead-claimant takeover |
| GitHub Actions | CI | Lint, tests with Redis+Postgres service containers |
Prerequisites: Python 3.11+, Docker (Redis + Postgres), git.
git clone https://github.com/<you>/idempotency-shield.git
cd idempotency-shield
docker run -d -p 6379:6379 redis:7
docker run -d -p 5432:5432 -e POSTGRES_USER=ledger -e POSTGRES_PASSWORD=ledger -e POSTGRES_DB=shield postgres:16
python -m venv .venv &&source .venv/bin/activate
pip install -e ".[dev]"
pytest # includes the 50-way storm and the crash-window test# use it in your own servicefromshield.middlewareimportidempotentfromshield.storeimportIdempotencyStorestore=IdempotencyStore(redis_url, key_ttl_s=86400, inflight_ttl_s=30)
@app.post("/payments")@idempotent(store, settings)asyncdefcreate_payment(request: Request, response: Response):
... # pair with UNIQUE(idem_key) in your table; ADR-002 explains whyMeasured in-process against the real app (2 vCPU container), benchmark/results/latency.json:
| Path | p50 | p95 |
|---|---|---|
| First-time request (claim + handler + Postgres insert + CAS) | 2.27 ms | 2.96 ms |
| Replay (store hit, no handler) | 0.65 ms | 0.83 ms |
| 50-way concurrent duplicate storm, full resolution | 73.8 ms | 116 ms max |
310 charges from 300 unique keys plus 10 storms of 50 duplicates each: exactly one charge per storm, asserted in the benchmark itself.
ADR-001: losers wait and replay rather than 409, and the measured cost of that kindness. ADR-002: why the shield refuses to claim exactly-once on its own, and where the second layer must live.
- Request-body fingerprinting (reject same key + different body as 422). Worth adding; deliberately kept out of v1 to keep the claim semantics auditable. Trigger: first real client bug from key reuse.
- Non-transactional side effects (emails, third-party calls). The crash window cannot be closed by a constraint there; the outbox pattern is the right tool and is its own project (
txn-exactly-once-ledger). - Framework ports (Spring, Express). The Redis contract (claim token, Lua CAS, TTL semantics) is the spec; ports are mechanical.
Keys are caller-chosen opaque strings; the store holds response bodies, so retention (default 24 h) is a data-governance knob exposed in config. In-flight TTL bounds lock lifetime, so a dead process can never wedge a key permanently. No secrets in code; connection strings from env.
| Failure | Detection | Behaviour | Recovery |
|---|---|---|---|
| Duplicate storm | By design | One handler execution; losers replay the winner | Tested at 50-way |
| Claimant dies mid-handler | In-flight TTL expiry | Waiter takes over the claim and executes | test_dead_claimant_takeover |
| Claimant dies after commit, before CAS | Retry re-executes the handler | DB unique constraint resolves to the existing charge | The war-story test; ADR-002 |
| Redis down | Claim errors | Endpoint fails closed (no unprotected execution) | Redis HA in prod; the guarantee never silently degrades |
| Winner slower than wait timeout | Waiter 409s after wait_timeout_s | Bounded waiting; client retries later and hits replay | Config knob |
| Stale claimant completes late | Lua CAS token mismatch | Completion rejected and logged, winner's response stands | complete_rejected log event |
The hardest part of this project was refusing to ship a lie that every demo of this pattern ships. The middleware works: storms collapse, replays are byte-identical, the happy path is airtight. Then I wrote the test I would want to see as a reviewer: commit the charge, wipe the idempotency record to simulate the claimant dying before completion, retry. The handler ran again. Without further defense that is a double charge, and no amount of Redis engineering fixes it, because a Postgres commit and a Redis write cannot be made atomic from the application.
The resolution is architectural honesty rather than cleverness (ADR-002): the shield's contract is replay and collapsing; at-most-once for the money lives in the ledger itself as UNIQUE(idem_key), with the handler converting the violation into "already done" and returning the existing charge id. The crash-window test now passes for the right reason, asserting both the response identity and the ledger count. The interview-ready generalization: idempotency is a property of the system, not of a layer, and each guarantee must be enforced in the store that can actually enforce it. Anyone whose middleware "guarantees exactly-once" has just not written this test yet.
- Request-body fingerprint check (same key, different payload: 422).
- Pub/sub wake-up replacing waiter polling for hot keys.
- Metrics: replay rate, storm size distribution, takeover count (each is a client-health signal).
- Spring Boot port of the store contract.
- First metric to watch in adoption: replay rate per client. A spike is a client retry-storm bug found before it becomes a support ticket.
MIT