Conversation
…X_HIP_GRAPH) Adds an opt-in GPU graph-capture path -- the HIP equivalent of CUDA Graphs. The sequence of GPU operations in the async (EP decode) eval body is CAPTURED once into a replayable graph, then REPLAYED as a single graph launch on later tokens. This collapses the ~1946 per-op host-side kernel dispatches of a decode step into one launch, cutting host launch overhead on the short-per-op decode path. Graph capture (a.k.a. CUDA/HIP Graphs) is a standard technique for launch-bound workloads; here it is exposed through HIP's graph API and gated by the env var MIGRAPHX_HIP_GRAPH (default OFF -> byte-identical eager path). It only engages on the sync-free async path (no trace/substitute callback, single context). Captures are keyed on sorted (param name, device ptr, shape) + queue, so identical buffers/shapes/stream replay safely; per-key output args are cached. context.hpp / gpu context provide capture_replay()/restore_queue() + a shared graph cache; program.cpp offers the eval body to the context on the async path. Co-Authored-By: Claude <noreply@anthropic.com>
pfultz2
reviewed
Sep 14, 2026
| // Defensive: handled but no cached outputs (should not happen) -> | ||
| // fall through to eager below. | ||
| } | ||
| } |
Collaborator
There was a problem hiding this comment.
Execution should be extended using operators not by changing the eval method, which is what #4956 already does.
rlegithub
marked this pull request as ready for review
September 14, 2026 21:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in GPU graph‑capture path — the HIP equivalent of CUDA Graphs — for the async (EP decode) eval body. The decode‑step op sequence is captured once into a replayable graph and replayed as a single graph launch on later tokens, collapsing the ~1946 per‑op host‑side kernel dispatches of a decode step into one launch (cuts host launch overhead on the short‑per‑op decode path).
Gated by
MIGRAPHX_HIP_GRAPH(default OFF → byte‑identical eager path);MIGRAPHX_HIP_GRAPH_DEBUGprints the WARMUP/CAPTURE/REPLAY trace. Only engages on the sync‑free async path (no trace/substitute callback, single context). Captures are keyed on sorted (param name, device ptr, shape) + queue, so identical buffers/shapes/stream replay safely; per‑key output args are cached. Correctness is preserved on any capture failure (falls back to eager).context.hpp/ gpu context providecapture_replay()/restore_queue()+ a shared graph cache;program.cppoffers the eval body to the context on the async path.Generic, opt‑in — not model‑specific (surfaced during GPT‑OSS‑20B decode).
Test plan
MIGRAPHX_HIP_GRAPH=1it engages (WARMUP → CAPTURE → REPLAY) and decodes correctly (~57 tok/s); default‑off is the eager path (no capture, 55.7 tok/s, byte‑identical).