Skip to content

fix(relay): emit BUZZ_SYNC_REQUIRED gap signal on fan-out backpressure drops - #6090

Open
mfethe1 wants to merge 1 commit into
block:mainfrom
mfethe1:fix/fanout-gap-signal
Open

mfethe1 wants to merge 1 commit into
block:mainfrom
mfethe1:fix/fanout-gap-signal

Conversation

@mfethe1

@mfethe1 mfethe1 commented Aug 17, 2026

Copy link
Copy Markdown

What

Relay-side fix for the sync-lag diagnosis finding that the first broken boundary in relay->client sync is silent fan-out frame drops under backpressure. When the priority ctrl channel's fan-out buffer is full, the relay now emits a gap signal to the affected connection instead of dropping frames invisibly, letting clients know their view is stale.

Contract (frozen)

  • Signal: ["BUZZ_SYNC_REQUIRED","backpressure"] on the priority ctrl channel.
  • Emitted only on fan-out drops (TrySendError::Full from the fan-out path). A closed ctrl channel cancels immediately; no signal into a closed channel.
  • sync_required(reason) is monomorphic over SyncRequiredReason::Backpressure — the relay cannot emit a reason the contract does not define.
  • Grace counter behavior preserved exactly as pre-existing.
  • buzz_fanout_dropped_frames_total counts every drop unsampled.
  • Disconnect counting is CAS-guarded (once_disconnect_counted) so concurrent disconnect paths count once per connection (8-thread regression test).
  • Protocol documentation lives in the protocol.rs module docs beside the RelayMessage definitions.
  • No client-side handling — relay-only by design; the client contract is frozen for a follow-up.

Diff

3 files / 736 ins / 27 del, all under crates/buzz-relay/ (handlers/event.rs, protocol.rs, state.rs). Single commit on current main (f956e6fe), no rebase.

Verification at exact head 0048e96

Two independent Windows (MSVC) hosts, separate gate stacks:

Gate Host A Host B (independent)
cargo test -p buzz-relay --lib --no-fail-fast 885 passed / 43 ignored / 3 failed 884 passed / 4 failed / 43 ignored
cargo fmt --all -- --check clean clean
cargo clippy -p buzz-relay --tests --all-features -- -D warnings clean clean

All failures are pre-existing host-shaped issues reproducing on the f956e6fe base:

  • api::git::policy::tests::bash_hmac_matches_rust_hmac and bash_hmac_single_ref — WSL bash resolution.
  • api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo — redis-gated.
  • (Host B only, intermittent) telemetry::tests::trace_context_lookup_does_not_enable_callsites — passes in isolation (-- --exact); the test exists unchanged at base (telemetry.rs:475) and this diff touches zero telemetry code, so the flake is scheduler-shaped (global tracing callsite registration leaking between parallel tests), not diff-shaped. A Host A rerun reproduced the same intermittent flake.

Per the repo's ci.yml path-filter semantics, a crates/** diff triggers rust-lint / unit-tests / backend-integration on ubuntu — maintainer-approved CI runs on this PR are the supported-environment complementary evidence.

Attribution: commits carry Co-authored-by + Signed-off-by for the human operator; no AI attribution trailers.

Evidence (contract 2026-08-17, tier T2 protocol/relay)

Locked-head exception applies: paired-gate receipts are locked at this exact head SHA, so this evidence lives in the body rather than a docs/pr-evidence/6090/ commit (a new commit would move the head and invalidate the receipts).

Before — problem reproduced at base f956e6fe (read-only diagnosis, 2026-08-16):

  • Production-relay probe (NIP-42-authenticated raw WS subscriber, 2026-08-16 19:02–19:11 UTC): healthy-path pushes arrive in 254–457 ms and are queryable on first poll — proving the reported lag is a loss branch, not latency.
  • Static trace at f956e6f: try_send_ws_message (state.rs:592-620) on TrySendError::Full DROPS the frame with a server-side-only warn!; drop_count is never signaled to the client. Below the grace limit (default 15, config.rs:566-580) the socket stays "connected" with permanent holes in its event stream.
  • Runtime consequence: a briefly backgrounded consumer sees events "late until restart"; events were persisted all along and only reappear via reconnect-replay (desktop relayReconnectReplay.ts, mobile relay_session.dart:261-262).
  • Diagnosis record + probe captures: RESEARCH/SYNC_LAG_DIAGNOSIS_LANE3_HANDOFF_2026_08_16.md, .scratch/sync-lag-probe/probe_long.log, exp2.log (agent workspace).

After — fix verified at head 0048e96a0 (git rev-parse HEAD in the same shell as the runs):

  • Runtime harness (local relay 127.0.0.1:3000, BUZZ_SEND_BUFFER=1, scratch DB buzz_gaptest; raw tokio-tungstenite + NIP-42 auth, victim stalled 8s while publisher floods kind-1): victim received 5 buffered EVENTs, then ["BUZZ_SYNC_REQUIRED","backpressure"] as the ctrl channel saturated at capacity 8 — the exact path that used to drop silently now emits the contracted client-visible gap signal. Capture taken during development at pre-fix head 3cf682b50; the delta to this head is the metrics-inflation fix below + commit trailers (no mechanism change to the signal path).
  • Wire contract pinned by the exact-bytes test (protocol.rs) and the end-to-end test through send_fanout_frames (handlers/event.rs) — both run green in the two-host gate table above at this exact head.
  • Every drop counted unsampled (buzz_fanout_dropped_frames_total); the harness also caught drops-after-cancel re-incrementing buzz_ws_backpressure_disconnects_total (observed 46 for one connection) — fixed via is_cancelled() idempotence guards with regression test repeated_drops_after_cancel_do_not_inflate_disconnect_counter, in this head.

End-to-end: the first broken boundary in relay→client sync (silent fan-out drops under backpressure) is no longer silent: the relay emits a contracted gap signal on the priority ctrl channel, letting clients know their view is stale without tearing down the socket. Client-side replay on the signal is deliberately OUT of this PR (frozen client contract; follow-up lane).

Verified-by: paired Bingo + Winnie gate at 0048e96a0 (two independent hosts, table above); gate-call PASS event 77628f7d (Bingo/Winnie paired authority, Michael tiebreak, Ernie concurrence 2026-08-17T00:37Z). Contract: agent-workspace GUIDES/PR_EVIDENCE_CONTRACT.md.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant