diff --git a/INDEX.md b/INDEX.md index 3803c73..941dd27 100644 --- a/INDEX.md +++ b/INDEX.md @@ -12,7 +12,7 @@ follow the cross-pointers in their index or take the next matching seeded domain | Domain | Status | Route here when | |--------|--------|-----------------| | [databases](wiki/databases/index.md) | **seeded** | Designing schemas/tables/keys, choosing or evaluating indexes, writing or optimizing queries, choosing transaction/isolation behavior, surveying live data to derive a rule, verifying additive migrations | -| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, call-site enumeration before a contract change, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, consuming LLM APIs (completion validation, context budgeting), consuming external-API responses, externally-owned defaults, object-storage references) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) | +| [backend](wiki/backend/index.md) | **seeded** | Server-side application code — language-agnostic (`common/`: API contracts, call-site enumeration before a contract change, idempotency, JWT, timeouts/retries, caching, jobs, transactions in app code, shared state/pools, errors, consuming LLM APIs (completion validation, context budgeting), consuming external-API responses, externally-owned defaults, object-storage references, sync-vs-async integration choice, WebSocket/SSE connection lifecycle) plus stack subtrees: `java/` (JPA, Spring proxies, JVM threads/memory), `node/` (event loop, promises, runtime validation, shutdown), `python/` (GIL/asyncio, pydantic, WSGI/ASGI workers, language traps) | | [frontend](wiki/frontend/index.md) | **seeded** | Web UI code: state placement, rendering performance, in-UI data fetching (races, infinite scroll), auth token handling, forms, XSS-safe output, accessibility | | [infrastructure](wiki/infrastructure/index.md) | **seeded** | CI/CD pipelines, secrets in build/deploy, container image builds, rollout/rollback strategy, observability (logs/metrics/alerting), per-environment/path-valued config, multi-agent orchestration (worker liveness signals, shared run state, tmux pane delivery, completion gates, worktree-isolated workers) | | [testing](wiki/testing/index.md) | **seeded** | Writing or structuring automated tests: level choice, cases/assertions, test data, mock decisions, flaky tests (release-process quality → qa) | diff --git a/log.md b/log.md index bd066cf..e2378d7 100644 --- a/log.md +++ b/log.md @@ -44,3 +44,4 @@ Append-only. Format: `## [YYYY-MM-DD] -` (reus exact name for later `send-prompt.sh`); the script also exports the guardrails escalation env into each worker. Trust-screen wording drifts between CLI releases — if a launch hangs, set `LO_READY_EXTRA` / `LO_TRUST_EXTRA` (substrings) or -`LO_READY_TIMEOUT`. +`LO_READY_TIMEOUT`. `status-update.sh` resolves the status file's `session` field +from `tmux display-message -p '#S'` only when the caller is itself inside tmux +(`$TMUX` set) — a coordinator-side call (this shell, not a worker's tmux pane) +must pass `STATUS_SESSION=` explicitly, or the record's `session` +field is left absent rather than guessed from whatever tmux session happens to +be active. `watch-status.sh` now exits **5** on a pending guardrails escalation (approve/deny, clear `.orchestration/escalations/`, then DELIVER the outcome to the now-idle worker with `scripts/send-prompt.sh send lo- "approved — re-run: , then @@ -479,7 +484,13 @@ exits — handle, then relaunch watch with the same target: recurs while `questions/.json` exists, like exit 5; exit 5 wins when both are pending): read the record (`{ts, taskId, question, options, worktree}`), answer with `scripts/send-prompt.sh send lo- ""`, delete the record - file, relaunch watch. + file. If the task's status was recorded `phase=failed` when it asked the + question, reset it to the phase you actually observe (read the worker pane + first) BEFORE relaunching watch — the reset IS a normal status write, not a + new phase word: `STATUS_DIR= STATUS_SESSION= sh + scripts/status-update.sh note="reset after exit-6 + answer"` — otherwise `watch-status.sh` counts the stale `failed` phase and + aborts with exit 3 again on the very next poll. Then relaunch watch. - **7 — stalled live worker** (prints `[watch] worker stalled — :`; the weakest signal — failed(3) and all-reached(0) win over it; driven by `tmux-worker-stalled.sh`, silence threshold `LO_STALL_SEC` default 600s; a diff --git a/skills/orchestrate/scripts/send-prompt.sh b/skills/orchestrate/scripts/send-prompt.sh index 2219153..5c69f89 100755 --- a/skills/orchestrate/scripts/send-prompt.sh +++ b/skills/orchestrate/scripts/send-prompt.sh @@ -35,6 +35,8 @@ # env: # LO_QUEUED_PATTERN substring meaning "queued behind a busy turn" # LO_BUSY_PATTERN substring meaning "mid-turn" (state only) +# LO_PASTED_PATTERN substring meaning "collapsed to an unsubmitted paste +# placeholder" (issue #96, default: [Pasted text) # LO_PANE_TAIL_LINES non-empty pane lines searched for those (default 6) # LO_CONFIRM_DELAY seconds to settle before classifying a send (default 1) # LO_PICKUP_TIMEOUT wait deadline in seconds (default 180) @@ -54,6 +56,12 @@ TMUX_BIN=$(command -v tmux) || { echo "send-prompt: tmux not found" >&2; exit 12 queued_pat="${LO_QUEUED_PATTERN:-Press up to edit queued messages}" busy_pat="${LO_BUSY_PATTERN:-esc to interrupt}" +# Issue #96: a send can collapse into an unsubmitted "[Pasted text #N ...]" +# placeholder left sitting in the input box — tmux still reports the send-keys +# call as successful. launch-session.sh's submit-confirm loop proved the +# remedy (a further Enter); cmd_send ports the same detection, bounded. +pasted_pat="${LO_PASTED_PATTERN:-[Pasted text}" + # Every grep against these uses `-- "$pat"`. A pattern beginning with '-' is # otherwise parsed as a grep flag ("unrecognized option"), which is the same # defect the '--' before the send-keys payload guards against. @@ -183,6 +191,30 @@ cmd_send() { note_pane "$1" echo "queued"; exit 4 fi + + # [Pasted text] guard (issue #96): the placeholder proves "buffered, not + # submitted" the same way queued_pat proves "buffered, not consumed" — so it + # gets the same retry-before-verdict treatment. Bounded to 3 extra Enters, + # mirroring launch-session.sh's submit-confirm loop; a pane still stuck after + # that is reported unconfirmed, never guessed delivered. + pasted_attempts=0 + while printf '%s' "$after" | grep -qF -- "$pasted_pat" 2>/dev/null \ + && [ "$pasted_attempts" -lt 3 ]; do + if ! "$TMUX_BIN" send-keys -t "$(target_pane "$1")" Enter; then + session_alive "$1" || { echo "gone"; exit 3; } + echo "send-prompt: send-keys failed for live session '$1'" >&2 + exit 6 + fi + pasted_attempts=$((pasted_attempts + 1)) + sleep "$confirm_delay" + after=$(pane_tail "$1") || { + echo "send-prompt: capture-pane failed for live session '$1'" >&2; exit 6; } + done + if printf '%s' "$after" | grep -qF -- "$pasted_pat" 2>/dev/null; then + note_pane "$1" + echo "unconfirmed"; exit 7 + fi + if [ "$after" != "$before" ]; then echo "delivered"; exit 0 fi diff --git a/skills/orchestrate/scripts/status-update.sh b/skills/orchestrate/scripts/status-update.sh index 36cba89..5863856 100755 --- a/skills/orchestrate/scripts/status-update.sh +++ b/skills/orchestrate/scripts/status-update.sh @@ -22,8 +22,14 @@ now=$(date -u +%Y-%m-%dT%H:%M:%SZ) # cross-platform (GNU/BSD) iso-8601 UTC wt=$(pwd -P) # physical path so loop-gate can match it # Record the tmux session name so watch-status can detect a dead worker. The # worker runs inside its tmux session; allow an explicit override (orchestrator -# / tests) via STATUS_SESSION. -sess="${STATUS_SESSION:-$(tmux display-message -p '#S' 2>/dev/null || true)}" +# / tests) via STATUS_SESSION. Only ask tmux when actually inside one ($TMUX +# set) — outside tmux, `tmux display-message` still exits 0 and answers with +# the server's most-recently-active session, which is not this caller's (#97). +if [ -n "${TMUX:-}" ]; then + sess="${STATUS_SESSION:-$(tmux display-message -p '#S' 2>/dev/null || true)}" +else + sess="${STATUS_SESSION:-}" +fi # Collect the extra key=value pairs into one JSON object first (in memory — no # file writes), so the whole record lands in a single atomic write below. diff --git a/tests/send-prompt.bats b/tests/send-prompt.bats index 4abcb0d..8bf99ba 100644 --- a/tests/send-prompt.bats +++ b/tests/send-prompt.bats @@ -228,6 +228,82 @@ mk_busy() { [ "$status" -eq 1 ] } +# ------------------------------------ send: [Pasted text] guard (issue #96) +# +# cmd_send must never report "delivered" while the prompt sits as an +# unsubmitted "[Pasted text #N]" placeholder. A real tmux pane cannot be +# driven through an exact multi-call capture-pane sequence (before -> after -> +# after each retry) deterministically, so these cases use a scripted fake tmux +# on PATH — same technique as the `keys` fake below, extended with a captures +# script so each successive capture-pane call returns the next scripted line. + +# $FAKE_CAPTURES holds one pane-snapshot per line; capture-pane returns the +# Nth line on its Nth call (clamped to the last line once exhausted), so a +# test can script exactly what cmd_send sees on the before-capture, the +# post-send capture, and each retry re-capture. +_use_fake_tmux_send() { + mkdir -p "$STUB_ROOT/bin" + FAKE_SEND_LOG="$STUB_ROOT/sends.log"; : > "$FAKE_SEND_LOG" + FAKE_CAPTURE_N="$STUB_ROOT/capture_n"; echo 0 > "$FAKE_CAPTURE_N" + FAKE_CAPTURES="$STUB_ROOT/captures"; : > "$FAKE_CAPTURES" + FAKE_ALIVE_FILE="$STUB_ROOT/alive"; : > "$FAKE_ALIVE_FILE" + export FAKE_SEND_LOG FAKE_CAPTURE_N FAKE_CAPTURES FAKE_ALIVE_FILE + cat > "$STUB_ROOT/bin/tmux" <<'FAKE' +#!/bin/sh +verb="$1"; shift +case "$verb" in + has-session) [ -e "$FAKE_ALIVE_FILE" ]; exit $? ;; + capture-pane) + n=$(cat "$FAKE_CAPTURE_N") + total=$(wc -l < "$FAKE_CAPTURES" | tr -d ' ') + idx=$((n + 1)) + [ "$idx" -gt "$total" ] && idx="$total" + sed -n "${idx}p" "$FAKE_CAPTURES" + echo $((n + 1)) > "$FAKE_CAPTURE_N" + exit 0 ;; + send-keys) + printf '%s\n' "$*" >> "$FAKE_SEND_LOG" + exit 0 ;; +esac +exit 0 +FAKE + chmod +x "$STUB_ROOT/bin/tmux" + PATH="$STUB_ROOT/bin:$PATH"; export PATH +} +_enter_count() { grep -cx -- "-t =$S: Enter" "$FAKE_SEND_LOG"; } + +@test "send: a [Pasted text] placeholder cleared by one retry is delivered" { + _use_fake_tmux_send + printf '%s\n' 'READY>' '[Pasted text #1 +2 lines]' 'T3_PASTE_RAN' > "$FAKE_CAPTURES" + run --separate-stderr sh "$SP" send "$S" 'echo T3_PASTE' + [ "$status" -eq 0 ] + [ "$output" = "delivered" ] + # submit Enter + exactly one retry Enter, never more once the placeholder clears + [ "$(_enter_count)" -eq 2 ] +} + +@test "send: a [Pasted text] placeholder that never clears is unconfirmed, not delivered" { + _use_fake_tmux_send + printf '%s\n' 'READY>' '[Pasted text #1]' '[Pasted text #1]' '[Pasted text #1]' '[Pasted text #1]' \ + > "$FAKE_CAPTURES" + run --separate-stderr sh "$SP" send "$S" 'echo T3_STUCK' + [ "$status" -eq 7 ] + [ "$output" = "unconfirmed" ] + # submit Enter + exactly 3 bounded retries — never an unbounded loop + [ "$(_enter_count)" -eq 4 ] +} + +@test "boundary: a placeholder alongside the queued indicator reports queued, not unconfirmed" { + # Order is unchanged: the existing queued_pat check still wins over the new + # placeholder guard, so no retry Enter is sent at all. + _use_fake_tmux_send + printf '%s\n' 'READY>' '[Pasted text #1] Press up to edit queued messages' > "$FAKE_CAPTURES" + run --separate-stderr sh "$SP" send "$S" 'echo T3_BOTH' + [ "$status" -eq 4 ] + [ "$output" = "queued" ] + [ "$(_enter_count)" -eq 1 ] +} + # ---------------------------------------------------------------- wait @test "wait: returns picked-up once the worker drains its queue" { diff --git a/tests/status-update.bats b/tests/status-update.bats index 16c7680..804853f 100644 --- a/tests/status-update.bats +++ b/tests/status-update.bats @@ -4,6 +4,18 @@ setup() { SU="${BATS_TEST_DIRNAME}/../skills/orchestrate/scripts/status-update.sh" export STATUS_DIR="${BATS_TEST_TMPDIR}/status" + # Session resolution reads $TMUX/$STATUS_SESSION from the ambient environment + # (issue #97: a suite launched from inside a real tmux session would + # otherwise silently inherit it and mask the non-tmux regression below). + unset TMUX STATUS_SESSION +} + +mk_tmux_stub() { # $1 = session name the stub's `display-message -p '#S'` answers with + d="${BATS_TEST_TMPDIR}/tmuxbin" + mkdir -p "$d" + printf '#!/bin/sh\necho "%s"\n' "$1" > "$d/tmux" + chmod +x "$d/tmux" + printf '%s' "$d" } @test "writes phase, timestamp, worktree and extras in one valid-JSON record" { @@ -36,3 +48,24 @@ setup() { ( cd "$BATS_TEST_TMPDIR" && STATUS_SESSION=lo-7 bash "$SU" t4 implementing ) [ "$(jq -r '.session' "$STATUS_DIR/t4.json")" = "lo-7" ] } + +@test "TMUX set, no STATUS_SESSION: asks tmux and records its answer (normal)" { + stub="$(mk_tmux_stub inside-session)" + run env TMUX=fake PATH="$stub:$PATH" bash "$SU" t5 implementing + [ "$status" -eq 0 ] + [ "$(jq -r '.session' "$STATUS_DIR/t5.json")" = "inside-session" ] +} + +@test "TMUX unset: tmux is never consulted, session field stays absent (regression, issue #97)" { + stub="$(mk_tmux_stub wrongly-resolved-session)" + run env PATH="$stub:$PATH" bash "$SU" t6 implementing + [ "$status" -eq 0 ] + [ "$(jq -r '.session // "MISSING"' "$STATUS_DIR/t6.json")" = "MISSING" ] +} + +@test "STATUS_SESSION wins outside tmux even with tmux on PATH (boundary)" { + stub="$(mk_tmux_stub wrongly-resolved-session)" + run env STATUS_SESSION=explicit PATH="$stub:$PATH" bash "$SU" t7 implementing + [ "$status" -eq 0 ] + [ "$(jq -r '.session' "$STATUS_DIR/t7.json")" = "explicit" ] +} diff --git a/tests/wiki-lint-prohibitions.bats b/tests/wiki-lint-prohibitions.bats index e5b5b0d..14a2a3e 100644 --- a/tests/wiki-lint-prohibitions.bats +++ b/tests/wiki-lint-prohibitions.bats @@ -18,11 +18,11 @@ setup() { # --- normal: the real corpus is already compliant --------------------------- -@test "real wiki: exits 0 with 0 violations and 61 directive units" { +@test "real wiki: exits 0 with 0 violations and 64 directive units" { cd "$REPO_ROOT" || return 1 run node "$CHECKER" wiki [ "$status" -eq 0 ] - [[ "$output" == *"directives: 61"* ]] + [[ "$output" == *"directives: 64"* ]] [[ "$output" == *"violations: 0"* ]] } diff --git a/wiki/backend/common/api-design/api-versioning-and-breaking-changes.md b/wiki/backend/common/api-design/api-versioning-and-breaking-changes.md new file mode 100644 index 0000000..4804a33 --- /dev/null +++ b/wiki/backend/common/api-design/api-versioning-and-breaking-changes.md @@ -0,0 +1,64 @@ +--- +id: backend-common-api-design-api-versioning-and-breaking-changes +domain: backend +category: api-design +applies_to: [general] +confidence: verified +sources: + - https://docs.stripe.com/api/versioning + - https://docs.stripe.com/upgrades + - https://datatracker.ietf.org/doc/html/rfc8594 +last_verified: 2026-08-17 +related: [backend-common-change-impact-call-site-enumeration] +--- + +# Versioning a Public API and Rolling Out Breaking Changes + +## When this applies + +An API has external callers you do not control (public API, mobile app in the +field, third-party integration) and you need to change its contract — add, +remove, rename, or retype a field or endpoint. Unlike +[backend-common-change-impact-call-site-enumeration] (internal callers you can +find and fix in the same change), an external caller cannot be enumerated or +force-updated, so the contract itself must carry the compatibility guarantee. + +## Do this + +| Change | Classification | Do | +|--------|-----------------|----| +| Add a new optional request parameter | Backward-compatible | Ship on the current version; no version bump needed | +| Add a new field to a response | Backward-compatible | Ship on the current version — callers must ignore unknown response fields (design new clients to tolerate unfamiliar fields from day one, not retrofit tolerance later) | +| Add a new endpoint or resource | Backward-compatible | Ship on the current version | +| Add a new event/webhook type | Backward-compatible | Ship on the current version — document that webhook consumers must not fail on an unrecognized event type | +| Remove or rename a field/endpoint; change a field's type or meaning; change validation to reject previously-valid input | Breaking | Requires a new API version; existing callers keep the old behavior until they explicitly opt in | +| Reordering fields in a response, or changing the length/format of opaque strings (IDs, error message text) | Backward-compatible | Ship on the current version — callers must not parse opaque strings positionally or assume a fixed length | + +## Deciding how to version + +| Choice | Do | +|--------|----| +| How clients select a version | Pick one mechanism and use it consistently: a request header (e.g. `Stripe-Version`) or a date/number embedded in the URL path. A header keeps the URL stable across versions; a path segment makes the version visible in logs/caches without inspecting headers | +| What a "version" identifies | A single version stamps the whole API's behavior for that request — not one flag per field. Stripe stamps every monthly release with the major version's name so a caller pinned to a version only ever receives backward-compatible additions until it explicitly upgrades | +| Deprecating an old version | Announce the deprecation and give a lead time before removal; send the `Sunset` HTTP header (RFC 8594) with the retirement date on responses from the version being retired, so client tooling can detect it without reading changelogs | + +## Edge cases + +| Case | Then | +|------|------| +| A field's value looks unchanged but the semantics changed (e.g. a status enum gains a new state the old client didn't expect) | Treat as breaking — the type is unchanged but the client's exhaustive switch/if-chain silently mishandles the new value; document new enum values as an explicit compatibility risk even though they are structurally additive | +| A caller needs to test a new version before committing | Support a rollback window (Stripe allows 72 hours) or a per-request version override, so an early adopter can revert without a second deployment | +| An internal caller and an external caller share the same endpoint | Version for the external caller's guarantee; the internal caller can be migrated directly at the call site instead ([backend-common-change-impact-call-site-enumeration]) | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Change a field's type/meaning on the existing version because "no one should be relying on that" | Ship the change under a new version and deprecate the old one with a `Sunset` date | You cannot enumerate external callers the way you can grep internal call sites — an assumption about who depends on a field is unverifiable | +| Remove an old API version the moment the new one ships | Keep both live through a deprecation window and signal it via the `Sunset` header | Callers need time to detect and act on the deprecation signal before the version actually stops responding | + +## Sources + +- https://docs.stripe.com/api/versioning — date-based versions, per-SDK version pinning, monthly vs major releases +- https://docs.stripe.com/upgrades — explicit backward-compatible change list, 72-hour rollback window +- https://datatracker.ietf.org/doc/html/rfc8594 — `Sunset` HTTP header field for signaling upcoming retirement diff --git a/wiki/backend/common/api-design/cors-and-preflight.md b/wiki/backend/common/api-design/cors-and-preflight.md new file mode 100644 index 0000000..da042d5 --- /dev/null +++ b/wiki/backend/common/api-design/cors-and-preflight.md @@ -0,0 +1,55 @@ +--- +id: backend-common-api-design-cors-and-preflight +domain: backend +category: api-design +applies_to: [general] +confidence: verified +sources: + - https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CORS + - https://fetch.spec.whatwg.org/#http-cors-protocol +last_verified: 2026-08-17 +related: [security-api-exposure-exposing-an-origin-http-api] +--- + +# Handling Browser CORS Requests and Preflight + +## When this applies + +A browser calls your API from a different origin (different scheme, host, or +port) via `fetch`/`XMLHttpRequest`, and the request fails in the browser +console with a CORS error even though a direct `curl` to the same endpoint +succeeds — CORS is enforced by the browser, not the server, so server-side +tools never reproduce it. Also applies when designing which endpoints need +`Access-Control-*` headers and whether the browser will send an OPTIONS +preflight before the real request. + +## Do this + +| Case | Do | +|------|----| +| Request is a "simple request" (GET/HEAD/POST, only CORS-safelisted headers, `Content-Type` is one of `application/x-www-form-urlencoded`, `multipart/form-data`, or `text/plain`) | No preflight is sent; the browser makes the request directly and only checks `Access-Control-Allow-Origin` on the response before exposing it to JS | +| Request uses a non-safelisted method (PUT/DELETE/PATCH), a custom header, or another `Content-Type` (e.g. `application/json`) | Browser sends an `OPTIONS` preflight first; the server must answer it with `Access-Control-Allow-Origin`, `Access-Control-Allow-Methods`, and `Access-Control-Allow-Headers` before the browser sends the real request | +| Public, unauthenticated endpoint (no cookies/credentials) | `Access-Control-Allow-Origin: *` is allowed | +| Endpoint reads cookies or `Authorization` and needs `credentials: 'include'` on the client | Respond with an explicit single origin (never `*`) in `Access-Control-Allow-Origin`, plus `Access-Control-Allow-Credentials: true` — the Fetch spec forbids the wildcard on any of `Access-Control-Allow-Origin`, `-Headers`, `-Methods`, or `-Expose-Headers` for a credentialed request, and the browser blocks the response client-side if you send it anyway | +| Allowlisting more than one origin | Do not join multiple origins with commas into one `Access-Control-Allow-Origin` value (browsers reject a header carrying more than one origin) — compute a per-request response: look up the request's `Origin` header against your allowlist and echo back only that single value, with `Vary: Origin` so caches don't serve one origin's response to another | +| Preflight fires on every request, adding a round trip | Set `Access-Control-Max-Age` (seconds) on the preflight response so the browser caches the result and skips re-preflighting the same method+headers+origin combination until it expires | + +## Edge cases + +| Case | Then | +|------|------| +| Preflight succeeds (200) but the real request still fails CORS | The preflight and the real response are checked independently — the real response also needs `Access-Control-Allow-Origin`; a proxy/CDN that strips CORS headers only from the real response is a common cause | +| A reverse proxy or API gateway sits in front of the origin | Confirm CORS headers are added at the layer that actually terminates OPTIONS — if the origin app never sees the preflight (the gateway auto-answers it), the origin's response headers still need to match or the real response is blocked | +| Non-browser client (server-to-server, mobile app, curl) reports a "CORS error" | It cannot — CORS is a browser-enforced restriction; the real failure is elsewhere (auth, network) and the report is misattributed | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Set `Access-Control-Allow-Origin: *` on an endpoint that also sets `Access-Control-Allow-Credentials: true` | Echo back the validated request `Origin` as a single explicit value | The Fetch spec's CORS protocol forbids the wildcard on a credentialed response, and browsers enforce this by blocking the response even if the server sends it | +| Debug a CORS failure by relaxing the server to allow every origin | Read the exact console error (it names the missing/mismatched header) and add only that header for the specific origins that need it | A blanket allow-all reopens the endpoint to any site's browser-side JS, including credentialed requests if cookies are involved | + +## Sources + +- https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CORS — simple vs preflighted requests, response headers, wildcard-with-credentials prohibition +- https://fetch.spec.whatwg.org/#http-cors-protocol — CORS-safelisted methods/headers, preflight algorithm, non-wildcard credentialed response requirement diff --git a/wiki/backend/common/architecture/sync-vs-async-integration.md b/wiki/backend/common/architecture/sync-vs-async-integration.md new file mode 100644 index 0000000..301f93b --- /dev/null +++ b/wiki/backend/common/architecture/sync-vs-async-integration.md @@ -0,0 +1,53 @@ +--- +id: backend-common-architecture-sync-vs-async-integration +domain: backend +category: architecture +applies_to: [general] +confidence: verified +sources: + - https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/sync-comm.html +last_verified: 2026-08-17 +related: [backend-common-jobs-idempotent-handlers, backend-common-reliability-timeouts-and-retries] +--- + +# Choosing Direct Call vs Queue vs Event Between Services + +## When this applies + +One service (or one module boundary) needs another service's work done, and +you're deciding whether the caller should call it directly and wait (sync), +hand it off through a queue for later processing (async request), or publish +a fact and let interested consumers react (event). This is a design decision +to make explicitly before writing the integration, not a default to fall +into because HTTP was the easiest thing to reach for. + +## Do this + +Decide by which property the interaction actually needs, not by habit: + +| Property you need | Choose | +|---------------------|--------| +| Caller needs the result before it can proceed (e.g. "is this payment authorized") | Synchronous call — the caller blocks and gets an immediate success/failure, so there is no ambiguity about whether the operation completed | +| Caller can proceed without knowing the outcome yet (e.g. "send a receipt email") | Asynchronous — hand the work to a queue/job and let the caller continue; the two systems are decoupled so the callee's downtime does not block the caller | +| Multiple independent consumers each need to react to the same fact, and the producer shouldn't know who they are | Event — the producer publishes what happened; consumers subscribe independently, and adding a new consumer requires no change to the producer | +| Caller cannot tolerate the callee being temporarily unavailable propagating into the caller's own failure | Asynchronous — a synchronous chain of calls fails the whole chain when any one link is down; an async handoff isolates the callee's downtime because the caller already returned | +| Strict consistency across both sides is required (both must succeed or both must roll back) | Synchronous, inside a single transaction boundary if possible — async introduces eventual consistency, which requires the caller to handle "the other side hasn't processed this yet" as a real state | + +## Edge cases + +| Case | Then | +|------|------| +| A "fire and forget" call is made synchronously only to avoid building a queue | Recognize this as a hidden coupling: the caller now blocks on (and fails with) a dependency it doesn't actually need a result from — move it to async once the queue infrastructure exists, don't leave it sync "for now" | +| An async handoff needs the caller to know it eventually succeeded (not just accepted) | Add a status the caller can poll, or a completion event the caller subscribes to — async does not mean the caller stops caring about the outcome, only that it stops blocking on it | +| Choosing between async-request (queue, one intended consumer) and event (pub/sub, unknown consumers) | If you can name every consumer today and the interaction is really "do this work for me," it's a queue, not an event; model it as an event only when the point is that consumers you don't control may subscribe later | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Default to a synchronous HTTP call because it's the simplest thing to write | Check whether the caller actually needs the result before proceeding, and whether the callee's downtime should be allowed to fail the caller | A sync call is the tightest coupling available — every synchronous hop in a chain becomes a shared point of failure and adds its latency to the caller's total latency | +| Make an integration async purely to "decouple things" without a durable queue behind it | Only claim the isolation/scalability benefits of async once the handoff is backed by a durable queue with retry semantics ([backend-common-jobs-idempotent-handlers]) | An in-memory or fire-and-forget "async" call without durability loses work on a crash and offers none of async's actual failure-isolation guarantee | + +## Sources + +- https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/sync-comm.html — consistency/latency/coupling/failure-isolation trade-off table between synchronous and asynchronous service communication diff --git a/wiki/backend/common/realtime/websocket-sse-lifecycle.md b/wiki/backend/common/realtime/websocket-sse-lifecycle.md new file mode 100644 index 0000000..f7226fa --- /dev/null +++ b/wiki/backend/common/realtime/websocket-sse-lifecycle.md @@ -0,0 +1,54 @@ +--- +id: backend-common-realtime-websocket-sse-lifecycle +domain: backend +category: realtime +applies_to: [general] +confidence: verified +sources: + - https://datatracker.ietf.org/doc/html/rfc6455 + - https://html.spec.whatwg.org/multipage/server-sent-events.html +last_verified: 2026-08-17 +related: [backend-common-reliability-timeouts-and-retries] +--- + +# Managing a Long-Lived WebSocket or SSE Connection + +## When this applies + +Building or reviewing a server-pushed realtime channel — WebSocket (bidirectional) +or Server-Sent Events (server-to-client only) — and deciding how the connection +authenticates, detects a dead peer, reconnects after a drop, and shuts down +without losing in-flight messages. A long-lived connection has a lifecycle a +normal request/response endpoint doesn't: it can silently die without either +side calling close, and it must survive a server restart. + +## Do this + +| Decision | Do | +|----------|----| +| Authenticating the connection | Authenticate once at connection/handshake time (WebSocket: during the HTTP upgrade; SSE: on the initial GET) — there is no per-message request to reattach auth to afterward, so a token that expires mid-connection needs an explicit re-auth or forced-reconnect path, not a check that silently stops enforcing | +| Detecting a dead peer (WebSocket) | Send ping control frames on an interval and require a matching pong within a timeout; a peer that stops responding to pings is dead even though the underlying TCP connection may still look open (common through NATs/load balancers that hold connections open past actual liveness) | +| Reconnecting after a drop (SSE) | Rely on the browser's built-in auto-reconnect, but set the server's `retry` field to control the delay, and always send an `id` field per event; the client automatically replays its last id via the `Last-Event-ID` request header on reconnect so the server can resume the stream instead of restarting it | +| Reconnecting after a drop (WebSocket) | Implement client-side reconnect with backoff explicitly — WebSocket has no built-in reconnect or resume, so the client must track its own last-known state and either replay a resume token or accept a fresh snapshot on reconnect | +| Backpressure (server sending faster than the client/network can drain) | Bound the per-connection outbound buffer; when it's full, drop or coalesce non-critical messages (e.g. keep only the latest of a repeated state update) rather than growing the buffer unboundedly, which turns a slow client into a server memory leak | +| Shutdown draining | On server shutdown/deploy, stop accepting new connections, send a close frame (WebSocket) or end the event stream (SSE) with enough lead time for clients to reconnect elsewhere, instead of dropping every open connection at once when the process exits | + +## Edge cases + +| Case | Then | +|------|------| +| A load balancer or proxy sits in front of the server | Confirm it forwards `Connection: Upgrade` for WebSocket and does not buffer the response for SSE (some proxies buffer by default, which delays every event until the buffer fills) | +| Client reconnects rapidly in a loop (e.g. auth keeps failing) | Apply exponential backoff with a cap and jitter on the client, and rate-limit reconnect attempts per client on the server, so a broken client cannot reconnect-storm the server | +| Ping/pong keepalive traffic itself becomes significant load at scale | Increase the interval rather than removing the check — the goal is bounded detection latency, not the shortest possible interval | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Assume a WebSocket connection is alive because the socket hasn't errored | Send ping/pong on an interval and treat a missed pong as dead | Intermediate proxies/NATs can hold a TCP connection open long after the actual peer has gone away, with no error surfaced to either side | +| Let the outbound buffer to a slow client grow to keep every message | Bound the buffer and drop/coalesce when full | An unbounded per-connection buffer against a slow or stalled client is a server-side memory leak that scales with the number of slow clients | + +## Sources + +- https://datatracker.ietf.org/doc/html/rfc6455 — ping/pong control frames, close handshake (close frame, status codes) +- https://html.spec.whatwg.org/multipage/server-sent-events.html — `retry` field, `id` field, `Last-Event-ID` reconnection header diff --git a/wiki/backend/index.md b/wiki/backend/index.md index 5689f57..fa04bae 100644 --- a/wiki/backend/index.md +++ b/wiki/backend/index.md @@ -5,7 +5,7 @@ three stack subtrees — route by concern first, stack second: | Subtree | Route there when | |---------|------------------| -| [common](#common-language-agnostic) (below) | The concern is language-agnostic: API contracts, enumerating call sites before a contract change, idempotency, JWT issuance, outbound calls, caching, jobs, transactions in app code, shared state/pools, exception structure, consuming LLM APIs (completion validation, context budgeting), consuming external-API responses, externally-owned defaults, object-storage references | +| [common](#common-language-agnostic) (below) | The concern is language-agnostic: API contracts, enumerating call sites before a contract change, idempotency, JWT issuance, outbound calls, caching, jobs, transactions in app code, shared state/pools, exception structure, consuming LLM APIs (completion validation, context budgeting), consuming external-API responses, externally-owned defaults, object-storage references, sync-vs-async integration choice, WebSocket/SSE connection lifecycle | | [java](java/index.md) | You are writing/reviewing JVM backend code (Java/Kotlin, Spring, JPA/Hibernate) and the concern is stack-specific: entity mapping, persistence context, proxy pitfalls, JVM threads/memory | | [node](node/index.md) | You are writing/reviewing Node.js/TypeScript backend code: event-loop blocking, promise error handling, runtime validation at boundaries, graceful shutdown | | [python](python/index.md) | You are writing/reviewing Python backend code: GIL/concurrency model, pydantic validation, WSGI/ASGI workers, language traps | @@ -26,6 +26,8 @@ Match your situation to a "load when" line; load only matching pages. | [idempotency](common/api-design/idempotency.md) | An endpoint with side effects (create, charge, send) can receive the same request twice — client retry after timeout, user double-submit, gateway retry; designing idempotency-key storage; deciding which operations are safe to retry | | [pagination-contract](common/api-design/pagination-contract.md) | Designing a list endpoint's request/response contract — cursor vs page-number, limit caps, total counts, expired-cursor behavior (the backing SQL/index → databases/query-optimization/keyset-pagination) | | [unenforced-declarations](common/api-design/unenforced-declarations.md) | Your system accepts declarative input (config file, DSL/manifest, policy block, schema annotation) and part of what a caller may write is unimplemented — an unknown key, a verb outside your vocabulary, or a knob recorded but never acted on; a user reports "I declared X and nothing happened"; choosing between reject/warn/ignore and where that strictness is selected | +| [cors-and-preflight](common/api-design/cors-and-preflight.md) | A browser-based caller on a different origin fails with a CORS error in the console (a direct curl to the same endpoint works); deciding whether a change to an endpoint's method/headers/content-type will trigger an OPTIONS preflight; designing `Access-Control-*` headers for a credentialed vs public endpoint; allowlisting more than one origin | +| [api-versioning-and-breaking-changes](common/api-design/api-versioning-and-breaking-changes.md) | An API has external callers you cannot enumerate or force-upgrade and you need to add/remove/rename/retype a field or endpoint; classifying a change as backward-compatible vs breaking; choosing a versioning mechanism (header vs URL); deprecating an old version (internal-only contract changes → [backend-common-change-impact-call-site-enumeration]) | ### change-impact @@ -99,3 +101,15 @@ Match your situation to a "load when" line; load only matching pages. |------|-----------| | [multi-object-write-ordering](common/storage/multi-object-write-ordering.md) | A diff writes two or more related objects (payload + checksum, data file + index entry, new version + the pointer that marks it current) with no transaction around the writes; reviewing such a diff for what a concurrent reader observes between the writes, or what a crash between them leaves behind | | [object-key-persistence](common/storage/object-key-persistence.md) | Persisting the result of an object-storage upload (`s3.upload()`, `lib-storage` `Upload`, a transfer manager) — choosing which response field goes in the DB column; building the read/signing path from a stored reference; migrating a column that holds URLs to keys; only large uploads 404 on read | + +### architecture + +| Page | Load when | +|------|-----------| +| [sync-vs-async-integration](common/architecture/sync-vs-async-integration.md) | Deciding whether one service/module should call another synchronously, hand work off through a queue, or publish an event — choosing by which property the interaction needs (immediate result, consistency, failure isolation, multiple independent consumers); a "fire and forget" call was made synchronously with no reason to block | + +### realtime + +| Page | Load when | +|------|-----------| +| [websocket-sse-lifecycle](common/realtime/websocket-sse-lifecycle.md) | Building or reviewing a WebSocket or Server-Sent Events channel — authenticating a long-lived connection, detecting a dead peer (ping/pong), reconnecting after a drop (SSE `retry`/`Last-Event-ID`, WebSocket client-side backoff), bounding backpressure on a slow client, or draining connections on shutdown/deploy | diff --git a/wiki/databases/index.md b/wiki/databases/index.md index c789ae8..fc3e9e9 100644 --- a/wiki/databases/index.md +++ b/wiki/databases/index.md @@ -46,6 +46,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [autovacuum-and-wraparound](operations/autovacuum-and-wraparound.md) | A write-heavy table bloats or slows over time; tuning autovacuum for a hot table; monitoring/preventing transaction-ID wraparound (age(datfrozenxid)); the database starts refusing writes to avoid wraparound; deciding VACUUM vs VACUUM FULL vs pg_repack | +| [data-backfill-migrations](operations/data-backfill-migrations.md) | A migration must populate/transform existing rows on a live table (not just alter the schema); choosing batch size and transaction boundaries so the backfill doesn't hold row locks for its full duration; making a backfill resumable/idempotent; verifying a backfill actually completed instead of trusting exit status | ## data-survey diff --git a/wiki/databases/operations/data-backfill-migrations.md b/wiki/databases/operations/data-backfill-migrations.md new file mode 100644 index 0000000..9535535 --- /dev/null +++ b/wiki/databases/operations/data-backfill-migrations.md @@ -0,0 +1,53 @@ +--- +id: databases-operations-data-backfill-migrations +domain: databases +category: operations +applies_to: [postgresql, general] +confidence: verified +sources: + - https://github.com/ankane/strong_migrations + - https://retool.com/blog/running-safe-database-migrations-using-postgres +last_verified: 2026-08-17 +related: [databases-schema-design-online-schema-changes, databases-schema-design-verifying-additive-migrations] +--- + +# Batching a Data Backfill on a Live Table + +## When this applies + +A migration needs to populate or transform existing rows on a table that +still takes production writes — backfilling a new column, converting a data +format, or copying values between columns — not just changing the schema. +[databases-schema-design-online-schema-changes] covers the DDL lock itself +(adding the column); this page covers writing data into the rows afterward, +which is the part that scales with table size instead of finishing instantly. + +## Do this + +| Decision | Do | +|----------|----| +| Backfill transaction shape | Never wrap the whole backfill in one transaction. A single-transaction backfill acquires a row-level lock on every row it touches and holds all of them until the entire backfill finishes, blocking any other write to those rows for the duration | +| Batching | Split the backfill into small batches (in the low hundreds to low thousands of rows), each committed in its own transaction, so each batch's row locks release immediately after that batch commits | +| Load on the primary | Add a short sleep between batches so the backfill does not saturate the connection pool or WAL throughput and starve normal traffic | +| Resumability | Write the backfill so re-running it (after a crash, a deploy, or a manual restart) produces the same result as one uninterrupted run — idempotent per batch (e.g. `WHERE new_col IS NULL`), not "resume from a remembered offset" that goes stale if rows are deleted or inserted mid-run | +| Verifying completion | Query the actual row count still matching the backfill's `WHERE` predicate (e.g. `new_col IS NULL`) after the run, not just "the job reported success" — a batch that errored partway through leaves unbackfilled rows with no other signal | + +## Edge cases + +| Case | Then | +|------|------| +| Backfill needs to run before the application can read the new column safely | Use dual-write first (the app writes both old and new representations) so the backfill only has to catch up historical rows, not race new writes — verify old vs new agree on a sample before cutting reads over | +| Table is large enough that a full-table `SELECT` for verification is itself expensive | Verify by comparing aggregate counts/checksums per batch as each batch completes, instead of one full-table scan at the end | +| Backfill touches a foreign-key-referenced table under concurrent inserts | Batch by primary-key range (not `OFFSET`/`LIMIT`) so concurrently inserted rows cannot shift which rows a later batch sees, which is what causes rows to be skipped or double-processed under `OFFSET` pagination | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Run `UPDATE ... WHERE ...` over the whole table in the migration's own transaction | Batch the update into small chunks, each in its own transaction, with a short sleep between | The whole-table update holds row locks on every touched row for the entire run, blocking concurrent writes to those rows | +| Treat "the migration script exited 0" as proof the backfill is complete | Query the row count still matching the backfill's target predicate after the run | A batch failure partway through a resumable-but-unmonitored script leaves silently unbackfilled rows | + +## Sources + +- https://github.com/ankane/strong_migrations — single-transaction lock-holding problem, `in_batches`, throttling between batches +- https://retool.com/blog/running-safe-database-migrations-using-postgres — small-batch/per-batch-transaction pattern, reentrant/idempotent backfill requirement diff --git a/wiki/infrastructure/deploy/feature-flag-lifecycle.md b/wiki/infrastructure/deploy/feature-flag-lifecycle.md new file mode 100644 index 0000000..a22043f --- /dev/null +++ b/wiki/infrastructure/deploy/feature-flag-lifecycle.md @@ -0,0 +1,56 @@ +--- +id: infrastructure-deploy-feature-flag-lifecycle +domain: infrastructure +category: deploy +applies_to: [general] +confidence: verified +sources: + - https://martinfowler.com/articles/feature-toggles.html +last_verified: 2026-08-17 +related: [infrastructure-deploy-rollout-and-rollback] +--- + +# Managing a Feature Flag from Introduction to Removal + +## When this applies + +Adding a flag/toggle to gate a feature — deciding what kind of flag it is, +how long it should live, and who removes it. [infrastructure-deploy-rollout-and-rollback] +covers the mechanics of a gated rollout (canary, health-gated promotion); this +page covers the flag itself as an artifact with its own lifecycle, since an +unmanaged flag outlives the reason it was created. + +## Do this + +| Flag category | Expected lifetime | Do | +|----------------|-------------------|----| +| Release toggle (hide incomplete work, ship trunk-based) | Days to a couple of weeks | Add a removal task to the backlog at the same time the flag is introduced — once the feature is fully rolled out, the flag and both code branches it guarded should be deleted in the same change | +| Experiment toggle (A/B test, cohort routing) | Until the experiment reaches statistical significance | Name it with the experiment, not the feature, so it's obviously tied to a decision that concludes; remove it (keeping the winning branch) once the experiment is decided | +| Ops toggle (kill switch, load-shedding) | Long-lived by design | Document it as a permanent operational control, not debt — but still name and inventory it so an incident responder can find it | +| Permissioning toggle (per-plan/per-tenant feature access) | Long-lived by design (can be years) | Treat it as a first-class authorization concern, not a temporary toggle — route the check through the same place other entitlement checks live, not an ad hoc flag lookup | + +## Deciding on cleanup + +| Situation | Do | +|-----------|----| +| A release or experiment toggle has been fully rolled out | Remove the flag and the losing code path in the same PR that confirms rollout — do not leave a "temporary" flag live past the decision that made it temporary | +| You don't know how many flags currently exist in the system | Keep an inventory (flag name, category, owner, creation date) — treat flags as inventory with a carrying cost, and periodically audit it against what's still gating live decisions | +| A flag is checked in more than a handful of places | That is a signal the flag has outlived a single-decision scope — split it (e.g. one config value read once at a boundary) rather than letting call sites accumulate | + +## Edge cases + +| Case | Then | +|------|------| +| Removing a flag would delete the *disabled* branch, but you're not fully sure it's safe | Flip the flag to fully-on in production first, observe, then remove the flag and dead branch as a separate, easily revertible change | +| A flag is checked inside a hot path (e.g. per-request) | Cache/precompute the flag's evaluated value per request or per deploy, rather than re-evaluating a remote flag service on every call | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Add a release/experiment flag with no plan for when it comes out | Add its removal to the backlog when you add the flag, or set an explicit expiration | Toggles multiply, and each live toggle multiplies the number of code paths that must be tested together | +| Let a release toggle become the permanent way a feature is enabled/disabled | Convert it to config/permissioning explicitly, or remove it and always-enable the code | A toggle meant to be transitional that never gets removed accumulates as untracked technical debt indistinguishable from a permanent flag | + +## Sources + +- https://martinfowler.com/articles/feature-toggles.html — flag category taxonomy (release/experiment/ops/permissioning), toggle-as-inventory framing, removal-task-on-introduction practice diff --git a/wiki/infrastructure/index.md b/wiki/infrastructure/index.md index 36a0771..6ae5ab1 100644 --- a/wiki/infrastructure/index.md +++ b/wiki/infrastructure/index.md @@ -52,6 +52,7 @@ Match your situation to a "load when" line; load only matching pages. | Page | Load when | |------|-----------| | [rollout-and-rollback](deploy/rollout-and-rollback.md) | Designing how a service reaches production (rollout strategy, health gating); preparing a risky release; a deploy involves a schema change, data migration, or feature flag and you need rollback mechanics | +| [feature-flag-lifecycle](deploy/feature-flag-lifecycle.md) | Adding a feature flag/toggle and deciding its category (release, experiment, ops, permissioning) and expected lifetime; a flag has outlived the rollout/experiment that created it; deciding when and how to remove a flag and its dead code path; a flag is checked in many places | ## observability