Uh oh!
There was an error while loading. Please reload this page.
feat(rpc,fusion): add ReconnectTimeout for remote compute calls with … - #166
Open
alexyakunin wants to merge 5 commits into
Open
feat(rpc,fusion): add ReconnectTimeout for remote compute calls with …#166alexyakunin wants to merge 5 commits into
alexyakunin wants to merge 5 commits into
Conversation
alexyakuninforce-pushed
the
feat/reconnect-timeout
branch
4 times, most recently
from
September 3, 2026 22:27
c3d22b0 to
fdf230aCompare…il once the peer stays away RpcCallTimeouts.ReconnectTimeout (default 0, per method via [RpcMethod(ReconnectTimeout = ..)]) is how long a call waits for a peer that lost its connection to reconnect - when the peer is disconnected at call time (and has been connected before; the very first connection is governed by ConnectTimeout alone) and when the connection drops while the call is in flight. A reconnect within the timeout lets the call proceed as usual, in-flight calls being resent as always. Once it elapses: - a remote compute call that has a cached value serves it as an unsynchronized computed, and the pending call - which carries the value's hash and stays registered for the resend - validates it once the peer is back: a "match" confirms it in place, a different result displaces it. This replaces the invalidate-on-reconnect of stale computeds, so a confirmed value no longer costs a recompute; ApplyRpcUpdate now waits for the connection with no timeout, since it always has a value to show; - any other call fails with RpcTimeoutException. In-flight calls are failed by RpcOutboundCallTracker.AbortOnReconnectTimeout, one loop per disconnect that wakes at the earliest deadline, so a connected peer pays nothing per call; calls issued while disconnected fail from RpcPeer.WhenConnectedOrReroute(RpcCallTimeouts, ..), which picks the tighter of ConnectTimeout and ReconnectTimeout. Every RPC timeout now throws RpcTimeoutException, a transient TimeoutException whose TimeoutKind (Connect, Run, Reconnect, Delay, Handshake, KeepAlive) tells which timeout fired; the plain TimeoutException instances Errors used to produce are gone. RpcOutboundCall.HasReconnectFallback exempts compute calls carrying a real cache entry from being failed; RpcPeer.HasEverConnected tells a reconnect from a first connection. RpcClientPeer.WhenConnectedOrReroute(timeout) fails at once when the next reconnect attempt is already scheduled past the deadline (ReconnectsAt), so a client that parks reconnects while offline does not sit the timeout out; the same applies to ConnectTimeout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Y92apva7VPeum3AxzAsBV
alexyakuninforce-pushed
the
feat/reconnect-timeout
branch
from
September 4, 2026 00:05
fdf230a to
246f445Compare…ML docs, small cleanups Explains the reconnect-timeout paths where they live (the two stale branches, the tracker loop, the client peer shortcut), documents every RpcCallTimeouts member with the RpcTimeoutKind its expiry reports, clamps the tracker wake-up delay, and simplifies RpcPeer.StopMode to a field-backed property. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Y92apva7VPeum3AxzAsBV
…out -> CacheFallbackDelay TimeSpan.MaxValue (= TimeSpanExt.Infinite) is now the only "no timeout" / "never" value, and zero always means "instantly". TimeSpanExt.AsTimeout() is the single validating converter for a configured TimeSpan / TimeSpan? / double (seconds) / double?: negatives and NaN throw, null and +Inf collapse to Infinite. ToShortString() prints "inf" / "-inf", and RpcCallTimeouts got a PrintMembers so it reads as "ConnectTimeout = inf, ..." rather than "10675199.02:48:05.4775807". ReconnectTimeout is gone. ConnectTimeout governs every wait for a connection, reconnect included - the RpcClientPeer.ReconnectsAt shortcut it already had applies unchanged - so a call issued while the peer is away no longer needs a second knob, and a call that was already sent is bounded only by RunTimeout, as it was before the feature. Removed with it: RpcTimeoutKind.Reconnect, Errors.ReconnectTimeout, RpcOutboundCallTracker.AbortOnReconnectTimeout, RpcOutboundCall.HasReconnectFallback, RpcPeer.HasEverConnected and the timeout-selecting WhenConnectedOrReroute(RpcCallTimeouts) overload. Its compute half survives as RpcCallTimeouts.CacheFallbackDelay: how long a remote compute call with a cached value waits for a disconnected peer before serving that value. Zero (the default) serves it at once, which is the behavior shipped since a1772cf; the call then stays registered, is resent, and its response confirms the served value in place ("match") or displaces it - no invalidate-on-reconnect. Also adds FusionRpcServeStalePeerChangeTest: a served stale value is confirmed or displaced by a *different* peer, since a peer change resends the pending call rather than reconnecting it. SwitchableRpcTestClient moved out of RpcUnsentCallTest so both suites share one harness. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Y92apva7VPeum3AxzAsBV
…onnect 99ac67a deleted RpcOutboundCallTracker.AbortOnReconnectTimeout along with ReconnectTimeout, which dropped the abort-on-disconnect behavior entirely - a sent call became bounded by RunTimeout alone. The machinery should have been kept and repointed at ConnectTimeout, which is what this restores: AbortOnConnectTimeout aborts a sent call once the peer stays away for ConnectTimeout, measured from the disconnect, failing it with RpcTimeoutException of Connect kind. RpcOutboundCall.HasCacheFallback (was HasReconnectFallback) comes back with it: a remote compute call holding a real cache entry is exempt, because the compute layer serves that value after CacheFallbackDelay and needs the call to stay pending so the resend can validate it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Y92apva7VPeum3AxzAsBV
…the call CacheFallbackDelay had its own timing logic inside the compute layer (a WhenDisconnected race plus WhenReconnectedChecked) while ConnectTimeout was an event the tracker delivered. Both are now deadlines owned by the single per-disconnect watcher, RpcOutboundCallTracker.HandleDisconnect, which only delivers them - what each means is up to the call: RpcOutboundCall.OnCacheFallbackDelay(delay) -> false by default RpcOutboundCall.OnConnectTimeout(timeout) -> fails the call by default RpcOutboundComputeCall overrides the first: with a real cache entry it raises RpcCacheInfoCapture.WhenCacheFallback (the compute<->call bridge that already carries CacheEntry and Call) and marks itself served, which exempts it from OnConnectTimeout - it has nothing left to fail, and its response is what validates the served value. With NoCache it declines, so the call falls straight through to OnConnectTimeout, whose error is transient and retried. IsCacheFallbackServed replaces the static HasCacheFallback: the exemption is now earned by actually serving something, not by merely being able to. ComputeRpc no longer races WhenDisconnected itself - it awaits WhenCacheFallback. Terminal peer errors still surface, via OutboundCalls.Abort completing the call. Tests for the two interactions the model introduces: CacheFallbackWinsOverConnectTimeoutTest (ReturnDefault + ConnectTimeout = 1: served, and alive well past the timeout) and NoCacheFallsThroughToConnectTimeoutTest (NoCache + CacheFallbackDelay = 0.3 + ConnectTimeout = 1: declined, then failed on the later deadline). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018Y92apva7VPeum3AxzAsBV
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…a cached value
RpcCallTimeouts.ReconnectTimeout (default 0, per method via [RpcMethod(ReconnectTimeout = ..)]) is how long a remote compute call that already has a cached value waits for a disconnected peer to come back before serving that value - both when the peer is disconnected at call time and when the connection drops mid-call. A reconnect within the timeout yields the fresh value. Otherwise the cached value is served as an unsynchronized computed, and the pending call - which carries the value's hash and is resent by the tracker on reconnect - validates it once the peer is back: a "match" confirms it in place, a different result displaces it. This replaces the invalidate-on-reconnect of stale computeds, so a confirmed value no longer costs a recompute. ApplyRpcUpdate now waits for the connection with no timeout, since it always has a value to show.
RpcClientPeer.WhenConnectedOrReroute(timeout) fails at once when the next reconnect attempt is already scheduled past the deadline (ReconnectsAt), so a client that parks reconnects while offline does not sit the timeout out; the same applies to ConnectTimeout.
Claude-Session: https://claude.ai/code/session_018Y92apva7VPeum3AxzAsBV