Conversation
StandaloneProcessClient's data-plane RPCs (BatchExists, BatchGetRanges, BatchPutRanges, RegisterMemory, etc.) constructed a bare grpc::ClientContext with no deadline, so a server-side stall on any of them blocked the calling scheduler rank forever with no way out -- reproduced directly under real long-context agentic load, where a BatchExists call for a large page-key set never returned and wedged the whole TP group via the collective all-reduce that follows lookup(). Add ArmDataPlaneDeadline(ctx), configurable via UMBP_DATA_PLANE_RPC_TIMEOUT_MS (default 10000ms), applied to the 18 routine calls. RegisterMemory gets its own, longer deadline (ArmRegisterMemoryDeadline / UMBP_REGISTER_MEMORY_RPC_TIMEOUT_MS, default 180000ms) since it is a one-time per-buffer setup call observed taking 90-120+ seconds per GPU against a large DRAM tier -- the routine 10s value false-fails it long before the server can finish. A deadline-exceeded status is handled identically to any other non-OK grpc::Status by callers (treated as "not found" / no-op), so no caller-side changes are needed. Validated on Kimi-K3 (MI355X, TP8, DCP8, MLA+KDA hybrid) under ~2000s of sustained real agentic traffic with zero RPC deadlocks -- a first after repeated multi-attempt deadlocks in the same batch_exists() call before this fix.
Every routine data-plane call in StandaloneProcessClient degrades a non-OK
grpc::Status to the same "not found" / no-op return used for a genuine miss
(cache hit rate quietly drops, nothing else changes). Before this, that made a
DEADLINE_EXCEEDED from the previous commit's ArmDataPlaneDeadline() completely
invisible -- indistinguishable in the logs from the key really not being
there. RegisterMemory's own MORI_UMBP_ERROR was the only reason today's
UMBP_SSD_ENABLED misconfiguration was findable at all.
Add LogDataPlaneRpcFailure(), which names DEADLINE_EXCEEDED explicitly
("timed out") versus any other non-OK status ("failed"), and call it at
every one of the 17 remaining silent call sites (Put, Get, Exists, BatchPut,
BatchPutWithDepth, BatchGet, BatchGetRanges, BatchPutRanges, BatchExists,
BatchExistsConsecutive, Clear, Flush, ReportExternalKvBlocks,
RevokeExternalKvBlocks, RevokeAllExternalKvBlocksAtTier, MatchExternalKv,
GetExternalKvHitCounts). DeregisterMemory already throws with the error
message attached and needs no change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
StandaloneProcessClient's data-plane RPCs (BatchExists,BatchExistsConsecutive,BatchGetRanges,BatchPutRanges,RegisterMemory, etc.) construct a baregrpc::ClientContextwith no deadline. The only deadline anywhere in this file isWaitReady()'s 500ms startup poll. A server-side stall on any data-plane call thereforeblocks the calling scheduler rank forever, with no way out.
Reproduced directly under real long-context agentic load on Kimi-K3 (MI355X, TP8,
DCP8, MLA+KDA hybrid): a
BatchExistscall for a large page-key set never returned,and
py-spyshowed all 8 TP ranks parked in the same frame indefinitely. Because_sync_restorable_prefixdoes a collective all-reduce right after the lookup, onerank's stuck call wedges the entire TP group.
Fix
ArmDataPlaneDeadline(ctx), backed byUMBP_DATA_PLANE_RPC_TIMEOUT_MS(default 10000ms), applied to the 18 routine data-plane call sites.
RegisterMemory(2 call sites:RegisterDeviceMemory,RegisterHostShmMemory) getsits own, longer deadline —
ArmRegisterMemoryDeadline/UMBP_REGISTER_MEMORY_RPC_TIMEOUT_MS(default 180000ms) — since it's a one-time,per-buffer setup call observed taking 90-120+ seconds per GPU against a large DRAM
tier. A single 10s deadline for all data-plane calls false-fails this one long before
the server can finish.
grpc::Statusby every caller (already treated as "not found" / no-op, not an exception) — verified
by reading the callers before writing this, so no caller-side changes are needed.
Validation
Ran Kimi-K3 (TP8, DCP8) against a standalone-process UMBP server under ~2000s of
sustained real agentic traffic with zero RPC deadlocks — the first run all day to get
this far without one, after repeated multi-attempt deadlocks in the same
BatchExistscall before this fix.