Conversation
…d bound Four review-gate items from PR block#4401 round 3 (Wren's binding hold at 0845a09, amended by Sami's alias and read-bound findings): 1. New buzz-media regression test: a real tower_http TimeoutError, wrapped the way axum wraps body errors, driven through process_video_upload's StreamReader read loop must surface as MediaError::RequestBodyTimeout / 408 — never Io / 500. Honest scope per Sami's mutation pass (M3): this pins the conversion chain, not the sniff/routing decision, which lives upstream in upload_blob and is stated as uncovered rather than implied-covered. Mutating either conversion arm (the read-loop TimedOut arm or the stream map's IdleTimeout arm) fails the test. 2. MEDIA_READ_TIMEOUT restored 300s: commit 4 silently tightened media reads 300s -> 60s while relaxing uploads (found by Sami). 300s preserves the bound reads already had and covers the multi-call pre-header path: the read handler awaits several sequential storage calls, each independently allowed up to 60s by rust-s3's per-call default, so a 60s request deadline could cancel a sequence whose individual calls are all within their own budgets. 3. Route-precedence regression test for the legacy /media/upload alias: the literal lives in the upload sub-router while /media/{sha256_ext} lives in the read sub-router, merged. If axum ever resolved the literal under the param capture, the alias would inherit the tight read wall-clock and the commit-3 regression would survive on exactly one route. A slow-but-progressing PUT /media/upload must complete under discriminating bounds (read deadline tighter than the upload duration), and GET /media/upload must answer 405 from the literal, not 408 from the param route's deadline. 4. Comment wording: "any duration"-shaped claims on the idle bound now qualify completion by the 3600s MEDIA_UPLOAD_CEILING instead of contradicting it. No production code changes beyond the MEDIA_READ_TIMEOUT constant and doc comments. Additive on 0845a09, no rebase. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
…d bound Four review-gate items from PR block#4401 round 3 (Wren's binding hold at 0845a09, amended by Sami's alias and read-bound findings): 1. New buzz-media regression test: a real tower_http TimeoutError, wrapped the way axum wraps body errors, driven through process_video_upload's StreamReader read loop must surface as MediaError::RequestBodyTimeout / 408 — never Io / 500. Honest scope per Sami's mutation pass (M3): this pins the conversion chain, not the sniff/routing decision, which lives upstream in upload_blob and is stated as uncovered rather than implied-covered. Mutating either conversion arm (the read-loop TimedOut arm or the stream map's IdleTimeout arm) fails the test. 2. MEDIA_READ_TIMEOUT restored 300s: commit 4 silently tightened media reads 300s -> 60s while relaxing uploads (found by Sami). 300s preserves the bound reads already had and covers the multi-call pre-header path: the read handler awaits several sequential storage calls, each independently allowed up to 60s by rust-s3's per-call default, so a 60s request deadline could cancel a sequence whose individual calls are all within their own budgets. 3. Route-precedence regression test for the legacy /media/upload alias: the literal lives in the upload sub-router while /media/{sha256_ext} lives in the read sub-router, merged. If axum ever resolved the literal under the param capture, the alias would inherit the tight read wall-clock and the commit-3 regression would survive on exactly one route. A slow-but-progressing PUT /media/upload must complete under discriminating bounds (read deadline tighter than the upload duration), and GET /media/upload must answer 405 from the literal, not 408 from the param route's deadline. 4. Comment wording: "any duration"-shaped claims on the idle bound now qualify completion by the 3600s MEDIA_UPLOAD_CEILING instead of contradicting it. No production code changes beyond the MEDIA_READ_TIMEOUT constant and doc comments. Additive on 0845a09, no rebase. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: Tal Weiss <major.tal@gmail.com>
230ac85 to
83f1d18
Compare
…d bound Four review-gate items from PR block#4401 round 3 (Wren's binding hold at 0845a09, amended by Sami's alias and read-bound findings): 1. New buzz-media regression test: a real tower_http TimeoutError, wrapped the way axum wraps body errors, driven through process_video_upload's StreamReader read loop must surface as MediaError::RequestBodyTimeout / 408 — never Io / 500. Honest scope per Sami's mutation pass (M3): this pins the conversion chain, not the sniff/routing decision, which lives upstream in upload_blob and is stated as uncovered rather than implied-covered. Mutating either conversion arm (the read-loop TimedOut arm or the stream map's IdleTimeout arm) fails the test. 2. MEDIA_READ_TIMEOUT restored 300s: commit 4 silently tightened media reads 300s -> 60s while relaxing uploads (found by Sami). 300s preserves the bound reads already had and covers the multi-call pre-header path: the read handler awaits several sequential storage calls, each independently allowed up to 60s by rust-s3's per-call default, so a 60s request deadline could cancel a sequence whose individual calls are all within their own budgets. 3. Route-precedence regression test for the legacy /media/upload alias: the literal lives in the upload sub-router while /media/{sha256_ext} lives in the read sub-router, merged. If axum ever resolved the literal under the param capture, the alias would inherit the tight read wall-clock and the commit-3 regression would survive on exactly one route. A slow-but-progressing PUT /media/upload must complete under discriminating bounds (read deadline tighter than the upload duration), and GET /media/upload must answer 405 from the literal, not 408 from the param route's deadline. 4. Comment wording: "any duration"-shaped claims on the idle bound now qualify completion by the 3600s MEDIA_UPLOAD_CEILING instead of contradicting it. No production code changes beyond the MEDIA_READ_TIMEOUT constant and doc comments. Additive on 0845a09, no rebase. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: Tal Weiss <major.tal@gmail.com>
83f1d18 to
dc2ead6
Compare
Signed-off-by: Tal Weiss <major.tal@gmail.com>
Git sends the same unauthenticated four-byte flush-packet probe before any smart HTTP POST whose body exceeds http.postBuffer — not just large pushes. Fetch negotiations cross the 1 MiB decoded threshold when the client's divergent local history contributes enough have lines (an in-limit repo plus ~21k unpushed divergent branches suffices; measured live at MAX_MANIFEST_REFS = 10_000 with divergent client history: 1.46 MiB decoded negotiation, probe fired, fetch got 401). Parameterize the receive-pack probe middleware over the service's (request MIME, result MIME) pair — the predicate was already otherwise service-agnostic — and mount it on the upload-pack POST route as well. All other predicate legs are unchanged: no Authorization / Content-Encoding / Transfer-Encoding headers, a single exact Content-Length of 4, and an exact 0000 body; near misses fall through to the normal authenticated path. Tests: the existing exactness/path-independence matrix now runs against both services (2 path shapes + 11 near-misses each), plus a cross-service case proving each route rejects the other service's request MIME. Fixes block#4423 Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: Tal Weiss <major.tal@gmail.com>
The relay previously bounded request *bytes* (RequestBodyLimitLayer) but never request *time*: a client that sent valid headers and then withheld the body parked a relay task indefinitely (measured live with 200 parked sockets). Three narrow deadlines close the measured seam: - API router: 60s tower_http TimeoutLayer (408 on expiry via with_status_code — tower-http's layer returns the configured status directly rather than surfacing Elapsed), ordered outside the body limit so a stalled body is cancelled by the deadline instead of sitting inside body-limit middleware. WebSocket routes are handshake bounded only; the established session escapes the response future after the 101 upgrade (proven by test). - Media router: 300s with the same ordering and semantics — bounds the future until response headers, not streaming response bodies. - Git compatibility probe: 60s bound on the pre-auth four-byte body collection, failing closed into the normal authentication path. No outer deadline is added around authenticated Git/CAS execution: PACK_OPS_TIMEOUT (300s) is not a total request budget — finalize_push and CAS publication carry additional deadlines, and an outer layer could cancel a valid push during publication. Git-wide budgeting and hyper header-read deadlines stay open on block#4424. Scope: core unauthenticated request-body surfaces plus the Git probe. The admin router, git policy router, SPA fallback, and health listener are unchanged. Tests inject millisecond deadlines (no production-constant sleeps): stalled body gets an empty 408 within the bound and never completes the handler; a dropped stalled future never invokes the handler (probe and router variants); an established WebSocket session survives the request deadline; probe near-misses still reach the normal path immediately. Addresses the minimal core of block#4424 Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: Tal Weiss <major.tal@gmail.com>
…timeout The 300s media TimeoutLayer bounded the whole request future, so any legitimate slow-but-progressing upload needing longer than 300s (a 500 MiB body below ~14 Mbit/s sustained) was cut off with a terminal 408 — the CLI does not retry 408. Found by Sami (RESEARCH/PR4401_MEDIA_DEADLINE_SLOW_UPLOAD.md). Uploads and reads are two timeout semantics, so the media router is split by route class and merged: - Upload routes get a RequestBodyTimeoutLayer(60s): the deadline resets on every body frame, so a progressing upload of any duration completes while a withheld body (the parked-task attack, block#4424) still fails closed — plus a generous 3600s wall-clock ceiling (matching the Blossom auth window) backstopping the post-body phase. The ceiling's trickle-permit cost is measured and documented on the constant. - Read routes (GET/HEAD /media/{sha256_ext}) carry no request body, so they keep a tight 60s wall-clock deadline — without it a hung storage read parks a task forever (measured by Sami: idle-only un-bounds every non-body stall). TimeoutLayer only covers until response headers, so streaming blob downloads are not truncated. The idle timeout surfaces as a body read error, not a synthesized response, so buzz-media gains classify_body_error: a typed check for tower_http::timeout::TimeoutError first, then the length-limit Display patterns. All three body-consumption paths in upload_blob map IdleTimeout -> new MediaError::RequestBodyTimeout (408), LengthLimit -> FileTooLarge (413), Other -> Io (500) — previously an idle trip surfaced as 500 pre-sniff (paging as a storage fault) or 413 on the non-video collect path. Verified live per TESTING.md: a paced 512 B/s upload (387s wall, exceeding the old 300s bound) completes and round-trips byte-exact; withheld upload body answers 408 at exactly 60s with the idle-deadline message; API withheld body still answers 408 at 60s; git push/clone/ ff-push/pull smoke passes at this tree. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: Tal Weiss <major.tal@gmail.com>
…d bound Four review-gate items from PR block#4401 round 3 (Wren's binding hold at 0845a09, amended by Sami's alias and read-bound findings): 1. New buzz-media regression test: a real tower_http TimeoutError, wrapped the way axum wraps body errors, driven through process_video_upload's StreamReader read loop must surface as MediaError::RequestBodyTimeout / 408 — never Io / 500. Honest scope per Sami's mutation pass (M3): this pins the conversion chain, not the sniff/routing decision, which lives upstream in upload_blob and is stated as uncovered rather than implied-covered. Mutating either conversion arm (the read-loop TimedOut arm or the stream map's IdleTimeout arm) fails the test. 2. MEDIA_READ_TIMEOUT restored 300s: commit 4 silently tightened media reads 300s -> 60s while relaxing uploads (found by Sami). 300s preserves the bound reads already had and covers the multi-call pre-header path: the read handler awaits several sequential storage calls, each independently allowed up to 60s by rust-s3's per-call default, so a 60s request deadline could cancel a sequence whose individual calls are all within their own budgets. 3. Route-precedence regression test for the legacy /media/upload alias: the literal lives in the upload sub-router while /media/{sha256_ext} lives in the read sub-router, merged. If axum ever resolved the literal under the param capture, the alias would inherit the tight read wall-clock and the commit-3 regression would survive on exactly one route. A slow-but-progressing PUT /media/upload must complete under discriminating bounds (read deadline tighter than the upload duration), and GET /media/upload must answer 405 from the literal, not 408 from the param route's deadline. 4. Comment wording: "any duration"-shaped claims on the idle bound now qualify completion by the 3600s MEDIA_UPLOAD_CEILING instead of contradicting it. No production code changes beyond the MEDIA_READ_TIMEOUT constant and doc comments. Additive on 0845a09, no rebase. Co-authored-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: npub1qyvc0c5kl4gqv2fd97fsk46tu378sqgy35vc83rvgfwne90sel7s0ed67d <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: Tal Weiss <major.tal@gmail.com>
dc2ead6 to
1b96b09
Compare
Signed-off-by: Tal Weiss <major.tal@gmail.com> # Conflicts: # crates/buzz-relay/src/api/git/transport.rs
🔐 Codex Security Review
|
|
@block/buzz-oss-team - poke on this one, gently. It is green and merges clean against main, so no workflow approval is needed just to look at it. Reason I am pinging instead of waiting: one of the three issues in here is #4424, where the relay bounds request bytes but never request time. Valid headers, then withhold the body, and the task parks forever. Unauthenticated. I sat 200 sockets on it to confirm. I also just rewrote the description. It was stale, still describing only the original large-push bug from back when that was all this was, which made the diff size look alarming for no reason. It now explains why three issues ended up in one PR (short version: same middleware, and each fix makes the next one necessary). If you would rather review the DoS part on its own, say so and I will split #4424 out. Either way works, I just do not want it sitting. |
wesbillman
left a comment
There was a problem hiding this comment.
Carl, an automated reviewer, commenting via Wes’s GitHub account.
Changes requested. Reviewed 4ab4f786085a23fe6126529861840eff6048ceee...36fab5314173da1d6c34f170ee6eccce41a22dcb.
P1: The new API wall deadline can strand committed events without audit or delivery
The inline finding describes a concrete postcommit cancellation path, not just an ambiguous timeout response. With audit backpressure, the new deadline cancels event processing after DB insertion but before audit enqueue and before fanout/workflow scheduling. Reposting the same event with fresh HTTP authorization returns duplicate: and does not repair the lost work. Ownership transfer, create-only community provisioning, and invite claim also await publication after committing; their retries are not a universal recovery mechanism.
I reproduced the event cancellation with a paused-clock standalone harness using the verbatim production timeout, audit-enqueue, and dispatch-scheduling functions. DB insertion/duplicate semantics and downstream delivery were stubbed, so this is a control-flow reproduction, not live relay/Postgres integration:
BASE: original=200 OK retry=200 OK stored=true audit=Some("00") dispatched=true
HEAD: original=408 Request Timeout retry=200 OK stored=true audit=None dispatched=false
Scope and smallest safe correction
The exact four-byte probe shim on both Git services is a narrow compatibility fix; I found no auth bypass or repository-data disclosure in that changed path. Its local PROBE_BODY_TIMEOUT already bounds the newly introduced unauthenticated read. The API/media-wide timeout policy is not required to make that probe safe. I recommend taking the offered split: keep the push/fetch compatibility shim and local body deadline here, and review #4424’s wider policy separately. That separate fix should bound request-body consumption without cancelling irreversible mutations, or provide tested durable/idempotent recovery. No new user-facing feature was found in the diff, but new global timeout semantics are meaningful scope expansion.
Media disposition
I did not establish a second blocking media defect. The 3600s whole-upload ceiling can still interrupt storage/publication after a valid body finishes near the deadline; Blossom auth is checked before those operations, not continuously through them. A missing sidecar leaves an unservable orphan and content-addressed retry can repair it. S3 cancellation/late-commit behavior was not independently reproduced, so I am recording this as a residual availability/orphan risk, not claiming corruption or an auth bypass. The idle timeout/error-classification changes themselves preserve the reviewed 408/413/500 distinction.
Validation and limits
- Inspected all ten changed files and traced Git auth/authorization, body-limit layering, API postcommit paths, media body/error/storage lifecycle, and retry behavior. Independent Git/router/media review lanes were consolidated rather than treated as a blanket safety certificate.
- The unchanged probe implementation and all four new probe unit tests passed in an isolated package with key dependencies pinned to the production versions. The cancellation harness reproduced the regression. Neither result is a full
buzz-relaybuild/test run. - I did not run the ignored live-relay/MinIO Git E2E or independently reproduce the author’s live push/fetch measurements. The current GitHub check snapshot does not show Rust build/test execution, and the security-review comment marks this exact range as requiring a fresh review. Green metadata checks are not evidence those validations ran.
Exit criteria: remove the outer API mutation deadline (the minimal split), or demonstrate cancellation-safe durable postcommit completion/retry through the real event and membership mutation paths. Retain the local Git probe deadline and obtain exact-head relay test plus live large-request workflow evidence before merge.
| .layer(TimeoutLayer::with_status_code( | ||
| StatusCode::REQUEST_TIMEOUT, | ||
| timeout, | ||
| )) |
There was a problem hiding this comment.
[P1] Do not time out the entire API mutation future
This helper is applied to /events and other writes, but their postcommit work is not cancellation-safe. ingest_event inserts the event (handlers/ingest.rs:3172-3204), then dispatch_persistent_event awaits the intentionally bounded audit queue before spawning fanout/workflows (handlers/event.rs:358-374,574-597). If the queue is full when this 60s wall deadline expires, TimeoutLayer drops that future: the DB event remains, but its audit entry and delivery/workflow task are lost. The same event retried with a fresh NIP-98 header exits as duplicate: at ingest.rs:3206-3211, before either can be repaired. This can occur under sustained audit pressure, or when a valid slow body consumes most of the total budget and only a short postcommit wait remains.
A paused-clock seam harness using the verbatim timeout and audit/dispatch functions reproduces base=200/audit+dispatch versus head=408/stored/no-audit/no-dispatch; DB storage is stubbed, not live integration. The new collector tests prove cancellation, not mutation recovery.
Bound body reception rather than the mutation service future, or split this unrelated API policy out while retaining the local Git probe timeout. Simply returning 408 after commit is not a safe failure mode here.
Yo good people, Tal again.
This started as one bug and ended up as three. Sorry about that. They are the same code path though, and I'll explain why:
The original one (#2880)
When a push is bigger than
http.postBuffer, Git first sends a tiny unauthenticated POST containing only the 4 byte flush packet0000. We answered401, so Git gave up before it ever sent the real authenticated pack. Net effect: large pushes were just impossible.The fix admits exactly that probe and nothing else. No
Authorization, noContent-Encoding, noTransfer-Encoding, exactly oneContent-Length: 4, body is exactly0000. Anything even slightly off falls back to the normal auth path.Then #4423
The same probe also fires before big fetches, not only pushes. You need an in-limit repo plus enough divergent local history to push the negotiation past 1 MiB. I measured it live: 1.46 MiB decoded, probe fired, fetch got
401.It is literally the same middleware, so fixing push and leaving fetch would have shipped the identical bug on the other verb. The predicate was already service agnostic, so I parameterized it over the (request MIME, result MIME) pair and mounted it on upload-pack too.
Then #4424, IMPORTANT!!!
Once we admit that probe, we are reading an unauthenticated request body. Turns out the relay bounds request bytes but never request time. Send valid headers, then simply don't send the body, and the task sits there forever. I parked 200 sockets like that to be sure.
So: the probe body collection gets a 60s bound that fails closed into normal auth, and the API and media routers get deadlines ordered outside the body limit, so a stalled body is cancelled by the deadline instead of sitting inside body limit middleware.
This is why it is one PR and not three. Landing #2880 alone means shipping an unauthenticated body read into a relay with no request time bound at all, and opening a follow up for the hole I just made reachable. That felt wrong.
The two media commits
My first attempt was a 300s wall clock timeout on the media router, and it was wrong. It killed legitimate slow uploads (a 500 MiB body under ~14 Mbit/s sustained) with a terminal
408, and the CLI does not retry408. Sami caught it.Upload routes now use an idle body timeout that resets on every body frame, so a slow but progressing upload finishes however long it takes, while a withheld body still fails closed. There is a 3600s ceiling behind it for the post body phase, matching the Blossom auth window. Read routes keep a tight wall clock deadline instead, because they carry no request body and a hung storage read would otherwise park a task forever.
Size, honestly
+1343/-54across 10 files. Most of that is tests:Content-Length: 4followed byTransfer-Encoding: chunked, and clones the bytes back byte for byteWhat I deliberately DID NOT DO
No outer deadline around authenticated Git/CAS execution.
PACK_OPS_TIMEOUTis not a total request budget,finalize_pushand CAS publication carry their own deadlines, and an outer layer could cancel a valid push mid publication. Git wide budgeting and hyper header read deadlines stay open on #4424.Also unchanged: admin router, git policy router, SPA fallback, health listener.
Validation
cargo clippy -p buzz-relay -p buzz-test-client --all-targets -- -D warningscargo fmt --all -- --check504flake in test(relay): mesh_demo echo test flaky under parallel suite — passes standalone, fails full-suite on clean main #2458 reproduces on clean main too, fix is fix(mesh): frame loss when recv_validated is cancelled mid-validation (#2458) #2745)401Happy to split #4424 out if you would rather review the security part on its own. I kept them together because the probe change is what makes the missing timeout reachable.
Fixes #2880
Fixes #4423
Refs #4424 (partial, see "what I deliberately did not do")