Huddles overhaul: drain-safe rooms, bounded send latency, typed relay errors, live route switching - #6585
Open
tlongwell-block wants to merge 13 commits into
Open
Huddles overhaul: drain-safe rooms, bounded send latency, typed relay errors, live route switching#6585tlongwell-block wants to merge 13 commits into
tlongwell-block wants to merge 13 commits into
Conversation
Co-authored-by: Max <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz> Signed-off-by: Max <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz>
tlongwell-block
force-pushed
the
huddles/overhaul
branch
from
August 22, 2026 23:56
8445c68 to
536d3c2
Compare
A relay drain (SIGTERM or owner-drain) cancelled every local owner client at once, so the last teardown emptied the room and took the archive + 48103 path: every pod restart ended every huddle it owned. The same path also ended a huddle the instant its last client blipped, so a reconnect found the channel archived. The leave path now distinguishes a drain-driven teardown (the owner draining token fired, or `shutting_down` / `owners.is_draining()` is set, covering single-pod mode where there is no mesh token) and only releases the lease and drops the empty room, leaving the channel alive for rejoiners to re-acquire through Redis. Ordinary last-leaver departures no longer end the room atomically. `remove_peer_and_check_idle` captures an `IdleGeneration` (the admission count) under the guard lock; a spawned task waits `ROOM_EMPTY_GRACE` and calls `end_if_idle`, which sets `ended` only if the room is still empty and no admission happened in between. A rejoin during the window fences the stale observation even if the rejoiner has already left again, and two leavers who both observed empty cannot both archive. Owner-loss or a drain that begins mid-window aborts the end; owner-loss wins a tie with the timer. The fence only holds if the rejoiner lands on the same `Room`. A pre-last leaver's `cleanup_if_empty` (or a failed pre-admission join) could run after the last leaver's idle observation and evict the empty room from the manager; the rejoiner would then get a fresh `Room` and the stale grace task would archive underneath it. The idle observation now sets `idle_hold` on the room and `cleanup_if_empty` refuses to evict while it is set. The hold survives `end_if_idle` so the room stays registered while `archive_channel` is in flight — a joiner in that gap meets `ended` on the same room instead of admitting into a fresh one whose pre-join DB check raced the archive. The hold clears on admission or via the fenced `release_idle_hold`, called after the archive resolves (success or rollback) and on the drain/abort paths that abandon the window. Tests: room idle generation (last-leaver-only, end-once, rejoin fences stale), manager eviction (concurrent-leaver ordering pins the room and the rejoiner fences the stale end; released hold permits eviction, stale release does not; ended room stays pinned and refuses joiners until the archive resolves; failed archive reopens and releases), and paused-time grace outcomes (holds through window, rejoin, no-rejoin ends, owner-loss aborts, mid-window drain aborts). Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz> Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Render reply events already present in the bounded channel window while the complete thread query loads. Keep the window data as placeholder-only so it does not delay the authoritative request. Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz> Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz>
Preserve typed relay admission codes through the audio connection boundary so UI copy and reconnect policy can act on production handshake failures. Move all active huddle playback consumers when the output route changes: remote peers, agent TTS, and STT acoustic-coupling policy. Co-authored-by: Max <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz> Signed-off-by: Max <d8473ee32b973aa31a21a65adddcc4b69cc2a8a4dee8121ecd51926e0cddbc02@buzz.block.builderlab.xyz>
tlongwell-block
force-pushed
the
huddles/overhaul
branch
from
August 23, 2026 00:16
536d3c2 to
4cff8ea
Compare
Squash of 4baccd5..b92854104 (Perci's agent-voice lane). Relay derives the publisher role from verified NIP-OA auth + active bot membership and elects one publisher seat per agent identity, same-pod and cross-pod; rosters and ordered deltas carry the role; desktop suppresses only the authoritative publisher's local fallback and drops the utterance on a typed election loss. Integration note: the lane's standalone `AudioConnectError` enum was folded into commit 6's typed `AudioRelayConnectError` as `is_lost_election()` (code == "duplicate_identity") so the audio handshake has one error type. `relay_api.rs` crossed the desktop 1000-line ratchet with the new tests, so its `tests` module moved to `relay_api_tests.rs` via `#[path]`, matching `stt.rs`/`models.rs`. Co-authored-by: Perci <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@buzz.block.builderlab.xyz> Signed-off-by: Perci <5a968df9a7494b4e019b9ecf739e088ba61097b4312124e9a88ae5b42e3f5f3e@buzz.block.builderlab.xyz>
An unexpected audio-relay disconnect used to emit
`huddle-audio-disconnected`, and HuddleContext.tsx answered with a
7-attempt loop (0/100/250/500/1000/2000/2000 ms, ~5.85 s total) that ended
in a silent `leaveHuddle()`. A relay deploy or a Wi-Fi roam that lasts
longer than that dropped the user out of the huddle with no explanation.
The audio pipeline task now awaits `reconnect::after_unexpected_disconnect`
in place of the event. `reconnect::run` redials for at least
`RECONNECT_WINDOW` (60 s) from the disconnect, with 250 ms → 5 s exponential
backoff jittered into [0.5, 1.0]; a `huddle_relay_draining` refusal redials
at the 250 ms floor because a replacement pod is coming. Huddle identity
(`is_current_huddle`) is re-checked after every dial and every sleep, so a
leave or a replacement huddle always wins and a socket won for the old
huddle is cancelled, never installed. Exhausting the window sets
`AudioLink::Lost` — still joined, visibly without audio — instead of leaving.
Progress is a new `HuddleState::audio_link` (`Live | Reconnecting{attempt,
draining} | Lost`), serde-tagged on `status`. It is a field, not a
`HuddlePhase` variant: `Connected | Active` is the liveness gate for
STT/TTS/agent voice, and a reconnecting phase would have stopped them
mid-blip. `claim_reconnect` flips `Live → Reconnecting` under the lock and
is the single guard against duplicate disconnect signals.
The `reconnect_huddle_audio` Tauri command and the React loop are deleted;
HuddleBar shows a muted pulsing "Reconnecting…" (or the relay-restart
variant) and a destructive "Couldn't reconnect audio" banner from the new
`audioLinkNotice`. Type mirrors in HuddleBar and the e2e bridge carry the
field.
`after_unexpected_disconnect` returns a boxed `Send` future because the call
graph is recursive (pipeline task → reconnect → `connect_audio_relay` →
pipeline task).
`connect_audio_relay` spawns the replacement pipeline before its handles
reach the reconnect loop. If that pipeline dies in the gap, its own
disconnect callback is refused by `claim_reconnect` (the link is already
`Reconnecting`), so the loop treats an already-cancelled token as a failed
dial and redials — `Live` never holds a dead sender.
Tests (paused tokio clock, injectable dial, each proven to fail when its
guard is removed): recovers after a 35 s outage without leaving, draining
redials promptly and is visible in state, leave during backoff exits the
loop, leave during an in-flight dial cancels the fresh socket, replacement
huddle is left untouched, a replacement pipeline that dies before install
is redialed not installed, duplicate signals do not start a second loop,
exhaustion marks Lost and terminates, not-live huddle is ignored, backoff
caps and jitters, `AudioLink` serialization. Node tests cover the banner
copy.
Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Brings in #6312 (mobile huddle participant polish) and #6610 (desktop audio protocol v2 downgrade: `parse_relay_frame` one-byte peer prefix, roster-presence occupancy check). Auto-merged cleanly in playout.rs and relay_api.rs; the branch's publisher-roster tracking and live output device switch sit alongside main's v2 frame parsing with no overlap. Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz> Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
tlongwell-block
marked this pull request as ready for review
August 23, 2026 15:34
Plain merge, no rebase, no force, per Tyler. Two conflicts resolved: - desktop/src-tauri/src/huddle/playout.rs: test-module collision. Main renamed continuous_dtx_does_not_extend_remote_floor_deadline to continuous_silence_does_not_extend_remote_floor_deadline (level-gated remote speech, #6610 follow-up); the branch added publisher_roster_uses_only_authoritative_role_and_tracks_transitions above it. Kept both: branch test intact, main rename adopted. - desktop/src/features/messages/useThreadReplies.test.mjs: main removed client-side aux hydration (#6572/#6447) and replaced the file with a source-contract test; the branch added a window-seeded-roots settle test for useThreadRepliesForRoots. Kept main contract test and re-based the branch settle test on the post-aux hook: dropped the deleted collectThreadAuxMessageIds/fetchAuxEventsByReference assertions, feed the raw invoke shape (next_cursor), and poll for settle so the assertion is not tick-sensitive under a loaded runner. Verified: desktop pnpm test 5572/5572 pass; src-tauri cargo test 2796 passed, 0 failed; cargo fmt/clippy clean on touched files. Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Preserve both sides of the e2eBridge import conflict. Adapt the new agent publisher role check to main's NIP-OA verifier contract by passing the verified auth event's signed creation time. No rebase or history rewrite. Remaining huddle review findings are not folded into this integration commit. Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz> Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
🔐 Codex Security Review
|
added 4 commits
September 2, 2026 11:25
Integrate main at 1c8321c. Automatic merge; no conflict resolutions or review repairs. Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz> Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Revert fd2cdab as a new commit. Remove relay roles, admission election and role-based fallback suppression as one feature. The positional postcard additions are incompatible across rolling upgrades; a wire-only revert would leave topology-dependent authority. Preserve baseline agent sockets, drain-safe rooms, reconnect, bounded send queue, output routing, typed errors and transcript improvements. Resolve conflicts by keeping newer auth and continuous-silence floor behavior. Baseline duplicate rendering remains a follow-up, not a partial guarantee. Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz> Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Pin every control variant and rejection ordinal, including nested multi-peer rosters and multibyte revisions, to the production encoder on main 1c8321c. Roundtrip tests alone cannot detect positional postcard schema changes. A deliberate peer-index/epoch order mutation fails this fixture. Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz> Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
Capture channel and huddle generation before dialing and carry that identity into disconnect recovery. Cancel pending recovery on leave, reset or a new huddle; keep transcription generations independent. Bound each complete dial to ten seconds and the remaining sixty-second recovery window, including backoff. Preserve stale/cancelled returned-socket fences. Unify encoder, sender, receiver and cancellation exits in one supervisor. Close the queue, abort all children, join only still-unjoined handles, then clear this connection's remote-human floor. Scope remote ownership by socket so old cleanup cannot erase a replacement or the local microphone floor. Log non-cancellation sibling JoinErrors during teardown. Retain checked roster parsing after publisher deferral. Add production-bound regressions for hung dials, cancellation boundaries, stale callbacks, task panics, joined child destruction and replacement-order cleanup; deliberate mutations demonstrate the guards fail closed. Keep routing behavior unchanged. Co-authored-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz> Signed-off-by: Meli <5aaa86bce934fc3445fc254aab560a40923f10252f92107e665073dede0e04d3@buzz.block.builderlab.xyz>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Huddles reliability overhaul
Status: repairs pushed, code-reviewed, CI green; live release approval withheld. The real two-client live pass is unrun. Local test-runner discrepancies remain disclosed below. Current head:
7927e0fa1c0aa97c238f06754c5d41ce3a455070.Included
Lostwhile remaining joined, rather than silently leaving.placeholderDatawhile authoritative fetch starts immediately; no freshness/cache-policy change.Publisher election deferred as one feature
New commit
f0a40531reverts the complete publisher-role/election/role-based fallback-suppression feature, preserving main’s existing agent sockets and behavior. Positional postcard schema additions were incompatible during rolling upgrades; reverting only mesh fields would leave topology-dependent authority. Main’s duplicate-rendering limitation remains. This PR does not promise exactly-once agent speech or publisher failover.8211976bpins production legacy control bytes (all variants, rejection ordinals, nested rosters); the fixture passes on main’s encoder and fails deliberate peer-index/epoch transposition. Future roles require explicit negotiated compatibility; the current stream hello has no extension negotiation, and membership capabilities need plumbing.7927e0faadds the bounded recovery and joined teardown repairs. All changes are new commits; main was merged as5880f377(main parent1c8321cd), with no rebase or force push.Verification ledger
7927e0fa: normal pre-push passed all eight lanes, including desktop JS 5,885 passed, Tauri 3,083 library tests passed / 19 ignored plus all integration targets, Tauri clippy, desktop typecheck/check, file-size and Rust tests. No hook bypass.cargo testat7927e0fatwice: 1,027 passed / 2 failed / 89 ignored. Failures: mesh-demo echo 504-vs-200 (reproduced on main) and telemetrytrace_context_lookup_does_not_enable_callsites(not yet attributed). Earlier restored-candidate full relay passed 1,029 + 13 integration tests; that earlier green does not supersede the cargo failures. Switching to full-package nextest process isolation at the same clean head passes 1,042/1,042, with 90 skipped and one test marked leaky. This is a test-interaction lead, not telemetry attribution; both runner results remain in the ledger.1c8321cdpassed 325/325. Independent trace found a missing mention separator (@quinnwith) in the DM failure and a self-targeted drag announcement in keyboard reorder; unchanged-main mention-settlement and dnd measurement sequencing are timing-race leads, not proven main-baseline attribution. No unrelated fixes included. Current-head CI subsequently passed all four smoke shards.Workspace receipts:
RESEARCH/PR6585_REPAIRS_2026_09_02.mdand its evidence directory, including frozen committed diff, main comparisons, mutation logs, normal push logs and preserved Playwright videos/screenshots/error contexts.Current-head CI — green
At
7927e0fa, all reported checks are terminal: 57 successful, 29 skipped, zero failures/pending. This includes Desktop Core, all four smoke shards, PostgreSQL, both desktop relay-backed integration shards and Relay E2E. CI run. CI closes the automated workflow gate; it does not erase local failures or substitute for audible two-client validation.Required live pass — not run
Two separate macOS clients/users, one managed agent, headphones and an authorized relay operator, built from the exact clean candidate:
Lost.Record client/relay commits, logs, recordings and restart receipts. Mock-browser tests cannot close this gate. Do not enable unsafe multi-pod staging audio to obtain a test.
Explicitly outside this PR
Typed Placement/shared codec cleanup, remote-human STT attribution, indicator roster/poll replacement, transport redesign/owner-routed TCP pipe, and Opus FEC/loss-hint work remain separate. Historical disappearing-transcript diagnosis was falsified; placeholder seeding is not presented as a fix for it.