Two-level workspace estimation for MatMulNBits (CUDA EP, fpA_intB/CUTLASS) by Copilot · Pull Request #29811 · microsoft/onnxruntime

Copilot · 2026-07-21T20:08:45Z

Description

Pilot for Phase A of the memory roadmap (#29775): give the CUDA EP a way to know a MatMulNBits node's transient CUTLASS scratch ("workspace") size before Run(), instead of relying on the accountant's blanket 1.5x heuristic that never sees the workspace term. Scoped to one kernel, in-tree only.

The workspace formula is extracted into one shared, stateless helper that all three call sites funnel through, so the numbers are identical by construction:

Shared formula (fpA_intB_gemm.h): ComputeFpAIntBGemmWorkspaceSize(m, n, k, sm, multi_processor_count) — pure arithmetic, no device/instance state. Overflow-checked end-to-end with SafeInt<size_t> (both SM90 and non-SM90 branches, including ceil-div numerators); returns nullopt on overflow rather than throwing. The 5 tile constants are promoted protected → public so the free function can read them. CutlassFpAIntBGemmRunner::getWorkspaceSize now delegates to it — byte-for-byte identical on valid inputs.
Eligibility single-source-of-truth (matmul_nbits.{h,cc}): CheckFpAIntBEligibility + EffectiveFpAIntBWorkspaceSm centralize the multi-part fpA_intB path decision (dtype, option/env gate, nbits/block/alignment, SM, prepacked/SM90 constraints). The constructor is refactored to call it (behavior-preserving), so Level 1 can never disagree with the runtime path.
Level 1 — partition-time (EstimateMatMulNBitsWorkspace, wired into CUDAExecutionProvider::GetCapability()): a plain op-type-dispatched free function (no virtual, no registration) that estimates from node attributes alone, before any kernel instance exists. Exposed via a slim forward-declaring header; log-only for this pilot (does not alter the accountant's budget number). Both the call site and the #include are double-guarded with #if !defined(DISABLE_CONTRIB_OPS) && defined(USE_FPA_INTB_GEMM).
Level 2 — instance-level (MatMulNBits::DeclareWorkspaceRequirements): new default no-op virtual on OpKernel. Because matmul_nbits.cc compiles into two hierarchies, the no-op is mirrored on both the in-tree and the plugin adapter OpKernel (following the PrePack precedent); the real CUTLASS-backed override is in-tree only (#ifndef BUILD_CUDA_EP_AS_PLUGIN). It feeds the same effective arch the runner resolves after setArch() (FpAIntBPackingSmForKernel(), not raw sm_), so it equals the runtime request when the queried shape matches.
Boundary-safe struct: WorkspaceRequirement (size_bytes, slot_id) lives in a new lightweight core/framework/workspace_requirement.h so both OpKernel hierarchies can include it without the adapter pulling in core/framework/op_kernel.h. This pilot uses one slot; the InlinedVector/slot_id generality is reserved for later multi-buffer kernels (e.g. Attention).

// getWorkspaceSize is now a thin delegate over the shared formula:
auto ws = ComputeFpAIntBGemmWorkspaceSize(m, n, k, sm_, multi_processor_count_);
ORT_ENFORCE(ws.has_value(), "fpA_intB workspace size overflow for m=", m, " n=", n);
return *ws;

Unit tests cover the pure helpers: formula exact values (non-SM90 + SM90), overflow/negative-dim nullopt, EffectiveFpAIntBWorkspaceSm (drift guard A), and CheckFpAIntBEligibility (drift guard C).

Motivation and Context

The workspace size is currently computed live inside ComputeInternal() (getWorkspaceSize(m, n, k)), so the partition-time budget accountant has zero visibility into it and falls back to a 1.5x multiplier on initializer + output bytes — which both under- and over-estimates, since the CUTLASS scratch is fp32 and unrelated to output size. MatMulNBits was chosen as the pilot because its formula is real, non-trivial, kernel-instance-independent, and single-path, and the op dominates quantized-LLM projection cost.

Notes for reviewers:

ABI: adding a virtual to in-tree OpKernel changes vtable layout — a known consideration for out-of-tree EPs compiled against older headers; the pilot does not solve it but flags it so it isn't discovered silently.
Not addressed by design: the plugin EP's separate budget loop still uses the 1.5x heuristic; substituting the estimate into the actual accept/reject decision (roadmap option (b)) and a real plugin-side override are follow-ups.
CUDA could not be built/run in this environment (no GPU); formula values were validated with a standalone host program and all changed files are clang-format clean.

azure-pipelines · 2026-07-21T20:08:54Z

Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

Pull request overview

Adds a two-level workspace estimation path for CUDA MatMulNBits (fpA_intB/CUTLASS) so workspace bytes can be computed deterministically (1) at partition time from node metadata and (2) at kernel-instance time from shapes, using a shared overflow-checked formula.

Changes:

Extract fpA_intB GEMM workspace sizing into a shared ComputeFpAIntBGemmWorkspaceSize() helper and route the CUTLASS runner through it.
Centralize fpA_intB eligibility + “effective SM for workspace” logic and reuse it from both the MatMulNBits constructor and the Level-1 estimator.
Introduce a new OpKernel::DeclareWorkspaceRequirements() default no-op (mirrored in the plugin adapter hierarchy) plus a lightweight WorkspaceRequirement POD; add unit tests for the pure helpers.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
onnxruntime/test/providers/cuda/test_cases/matmul_nbits_workspace_test.cu	Adds host-only unit tests for eligibility, effective-SM mapping, and the shared workspace formula.
onnxruntime/core/providers/cuda/cuda_execution_provider.cc	Logs a Level-1 MatMulNBits workspace estimate during `GetCapability()` (pilot, log-only).
onnxruntime/contrib_ops/cuda/quantization/matmul_nbits.h	Declares shared eligibility + Level-1 estimator API and wires constructor through the shared decision.
onnxruntime/contrib_ops/cuda/quantization/matmul_nbits.cc	Implements shared eligibility, Level-1 estimator, and Level-2 `DeclareWorkspaceRequirements()` override.
onnxruntime/contrib_ops/cuda/quantization/matmul_nbits_workspace_estimate.h	Adds a slim forward-decl header for calling the Level-1 estimator from core CUDA EP code.
onnxruntime/contrib_ops/cuda/llm/fpA_intB_gemm/fpA_intB_gemm.h	Adds the shared overflow-checked `ComputeFpAIntBGemmWorkspaceSize()` helper.
onnxruntime/contrib_ops/cuda/llm/fpA_intB_gemm/fpA_intB_gemm_template.h	Makes `getWorkspaceSize()` delegate to the shared helper.
include/onnxruntime/ep/adapter/op_kernel.h	Mirrors the new `DeclareWorkspaceRequirements()` virtual default no-op for the plugin adapter hierarchy.
include/onnxruntime/core/framework/workspace_requirement.h	Introduces a lightweight `WorkspaceRequirement` POD shared across both OpKernel hierarchies.
include/onnxruntime/core/framework/op_kernel.h	Adds the new `DeclareWorkspaceRequirements()` virtual default no-op on the in-tree `OpKernel`.

Ti-Tai Wang (titaiwangms) · 2026-07-21T20:52:53Z

Ran this PR through a 5-pass review (readability / correctness / adversarial / spec-adherence / cross-module integration). No Critical findings — the core design (shared ComputeFpAIntBGemmWorkspaceSize formula, CheckFpAIntBEligibility, effective-arch consistency, plugin-adapter no-op, SafeInt overflow handling) is a faithful, well-traced implementation of #29810, and the arch-consistency claim (Level 1 == Level 2 == runtime) was independently hand-verified end-to-end through setArch()/sm_ and holds for every case.

That said, there are a few things worth addressing before merge:

Major

The PR's core acceptance criterion — "Level 1 estimate == Level 2 estimate == actual runtime workspace request" — is not actually tested. matmul_nbits_workspace_test.cu only unit-tests the three pure building blocks (ComputeFpAIntBGemmWorkspaceSize, EffectiveFpAIntBWorkspaceSm, CheckFpAIntBEligibility) in isolation. Nothing calls DeclareWorkspaceRequirements() on a constructed kernel, calls EstimateMatMulNBitsWorkspace() on a real Node, or exercises LastComputeWorkspaceBytes(). Since this instrumentation hook exists specifically to let a test prove L2 == runtime, and the whole point of this pilot is to demonstrate that kernel-level workspace prediction is accurate, this is the one gap that keeps the PR from actually self-validating its own premise (right now the equivalence is proven only by static code tracing, not by an executed test). Recommend adding the model-based tests the issue's Test A/B described: build a single-node MatMulNBits model, Initialize(), then assert EstimateMatMulNBitsWorkspace() (Level 1) == DeclareWorkspaceRequirements() (Level 2) == LastComputeWorkspaceBytes() after running Compute() once, for both a representative fpA_intB-eligible config and a fixed-shape case.
EstimateMatMulNBitsWorkspace (Level 1) and the constructor's eligibility check can disagree due to session config precedence. The constructor resolves ep.cuda.fpa_intb_gemm via ResolveFpAIntBConfigOrEnv (session config first, env var fallback), but Level 1 reads only the ORT_FPA_INTB_GEMM env var. For an unprepacked weight enabled via session config but not the env var (or vice versa), Level 1 and the real kernel can disagree about whether the node takes the fpA_intB path. Currently harmless since Level 1 is log-only, but this should at least be called out in a comment (or ideally: plumb the resolved session option into GetCapability(), which does have access to ConfigOptions), since it will become a real under/over-estimation bug the moment the Level-1 number is used for an actual budget decision.
EffectiveFpAIntBWorkspaceSm() (Level 1) and FpAIntBPackingSmForKernel() (Level 2/runtime) implement the identical condition independently rather than one delegating to the other, relying on a comment ("if this changes, that must change too") to keep them in lockstep instead of the compiler enforcing it. Given the whole design intent here is "single source of truth so Level 1/2 can't drift," recommend making FpAIntBPackingSmForKernel() simply forward to EffectiveFpAIntBWorkspaceSm(sm_, weight_prepacked_).

Minor

LastComputeWorkspaceBytes() is only updated on the CUTLASS-GEMM branch of ComputeInternal(); a subsequent call that takes the GEMV or non-fpA_intB path leaves it stale. Worth a doc note or a reset at the top of ComputeInternal().
FpAIntBEligibility echoes N/K/nbits/block_size/weight_prepacked/has_g_idx back to callers, but only .eligible is ever read at either call site — either drop the unused fields or note in the struct comment why they're kept.
The block_size != 0 conjunct in CheckFpAIntBEligibility's base_ok is unreachable dead code (the preceding clause already restricts block_size to {32, 64, 128}).
matmul_nbits_workspace_test.cu lives under test/providers/cuda/test_cases/ (a CUDA-only test target), but the delivered tests are pure host-side functions requiring no device — worth confirming this is intentional rather than accidentally excluding them from CPU-only CI.

Happy to take another pass once these are addressed, especially #1 — that's the one that would let this PR demonstrate its own value empirically rather than by static argument.

…o-level estimation Major 1: Add MatMulNBitsWorkspace.EndToEndWorkspaceAgreement, a GPU-requiring test that builds a single-node MatMulNBits model (fp16/int4/block_size=32, N=256/K=1024), runs it through a real InferenceSession on the CUDA EP, and asserts that the partition-time estimate (EstimateMatMulNBitsWorkspace), the kernel-instance estimate (DeclareWorkspaceRequirements) and the actual runtime request (LastComputeWorkspaceBytes) are all exactly equal (proven: 1792 == 1792 == 1792). The test lives in a new core-world .cc bridged to the provider world by a slim probe header, so a full session and provider internals can coexist. Also exports onnxruntime_provider_test symbols (ENABLE_EXPORTS) so the dlopen'd cuda_ut module can resolve InferenceSession at load time. Major 2: Document the known Level-1/constructor eligibility divergence (session-config ep.cuda.fpa_intb_gemm vs env var). Fully fixing it needs ConfigOptions threaded through IExecutionProvider::GetCapability(), a cross-EP interface change; Level 1 is log-only so the divergence is safe. Documented in EstimateMatMulNBitsWorkspace. Major 3: FpAIntBPackingSmForKernel() now delegates to the shared EffectiveFpAIntBWorkspaceSm(), so the compiler (not a comment) guarantees Level 1 and Level 2/runtime resolve the effective arch identically. Minor 4: Fix build-macro guards from defined(USE_FPA_INTB_GEMM) to USE_FPA_INTB_GEMM (value check) in matmul_nbits_workspace_estimate.h and cuda_execution_provider.cc. Minor 5: Document LastComputeWorkspaceBytes() staleness. Minor 6: Remove unused FpAIntBEligibility struct; CheckFpAIntBEligibility returns bool. Minor 7: Remove dead block_size != 0 conjunct. Minor 8: Update test file-header comments. Also fixes a pre-existing build blocker: EstimateMatMulNBitsWorkspace's get_attr lambda used NodeAttributes::find()/it->second, which the provider-bridge IteratorHolder does not support; switched to range iteration. Renamed the pure-function test from .cu to .cc (it has no device code) to avoid nvcc choking on gtest's re2-backed regex support.

…precise E2E comment - cmake: WIN32-gate link onnxruntime_providers_cuda_ut against the onnxruntime_provider_test import lib so InferenceSession symbols resolve at link time on Windows (MODULE libs can't have unresolved externals). - cmake: add matmul_nbits_workspace_test.cc and matmul_nbits_e2e_workspace_test.cc to the plugin-mode explicit test list so they run in BUILD_CUDA_EP_AS_PLUGIN CI. - test: clarify e2e comment/scope that Level 1 calls the estimator directly, not a full GetCapability()-driven partition-time run.

Tianlei Wu (tianleiwu)

The shared workspace formula, overflow handling, effective-SM routing, and current-head macro guards look consistent for the in-tree CUDA EP. I am requesting changes for the plugin test integration: the newly added tests depend on the concrete in-tree MatMulNBits kernel and cannot safely run through the plugin wrapper as currently implemented.

…der_api.h convention MSVC treats the struct/class keyword mismatch as warning C4099, which is escalated to a hard error (C2220) under the CUDA EP's /W4 all-warnings-as-error build flags on Windows. onnxruntime::Node is forward-declared as 'struct' in core/providers/shared_library/provider_api.h; this header used 'class', causing Windows GPU CUDA CI and Windows GPU Kernel Documentation Validation to fail while Linux builds (used for all local GPU verification) were unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ade1ab12-1137-46bb-a92b-e730a6f28bf4

matmul_nbits_workspace_test.cc and matmul_nbits_e2e_workspace_test.cc directly construct the concrete in-tree CUDAExecutionProvider / MatMulNBits<T> kernel types. In a plugin build (BUILD_CUDA_EP_AS_PLUGIN), the provider lives behind the plugin DLL and SessionState::GetKernel() returns a core PluginEpOpKernel whose DeclareWorkspaceRequirements() is the default no-op (the MatMulNBits override is compiled out). Downcasting that wrapper via GetMatMulNBitsLastComputeWorkspaceBytes()'s static_cast is undefined behavior, and depending on linkage this can instead fail earlier on unresolved provider-internal symbols. This reverts the round-2 fix's addition of these sources to onnxruntime_test_providers_cuda_plugin_internal_test_src, per #29811 review feedback. Re-adding them requires first bridging DeclareWorkspaceRequirements through OrtKernelImpl for plugin kernels (tracked as Phase B / future follow-up work), not exercising the in-tree concrete-kernel downcast against a plugin session. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ade1ab12-1137-46bb-a92b-e730a6f28bf4

Implements the three remaining Test-plan cases from issue #29810 (Tests A/B were already covered by EndToEndWorkspaceAgreement). All are additive, test-only changes in matmul_nbits_e2e_workspace_test.cc; no production code changed. Test C - FixedShapeViaFreeDimensionOverride: input A declared ["seq", K]; SessionOptions binds free dim "seq"->512 before Initialize(). Proves the fixed override value flows through the NodeArg shape into Level 1 (estimator) and Level 2 (DeclareWorkspaceRequirements) and matches the real runtime request (all == 3584 bytes for M=512). Also asserts requirements[0].slot_id == 0. Test D - DynamicShapeNoOverrideFallsBack: input A declared ["seq", K] with no override. Level 1 returns nullopt (symbolic leading dim), Level 2 returns empty requirements (unknown dim -> fallback), and Run() still succeeds via the live GetScratchBuffer path (runtime request 1792 bytes). Test E - NonMatMulNBitsKernelDeclaresNoWorkspace: a control Add kernel (which does not override DeclareWorkspaceRequirements) hits the OpKernel base-class no-op -> Status::OK() and an empty requirements vector (verified it clears a pre-populated vector rather than appending). Tests C/D GTEST_SKIP on GPUs with compute capability < 7.5 (matching the production CheckFpAIntBEligibility gate) so they stay portable across CI GPUs, and derive the Level-2 TensorShape from the actual (possibly-overridden) NodeArg proto via the production onnxruntime::utils::GetTensorShapeFromTensorShapeProto converter, exercising the real TensorShapeProto -> TensorShape path (including dim_param -> -1) rather than a hand-built stand-in. Refactors BuildMatMulNBitsModelBytes to optionally declare A's leading dim as a symbolic dim_param (for Tests C/D) and adds a tiny fp32 Add model builder (Test E).

…2E test Round-4 review fixes on top of the MatMulNBits two-level workspace-estimation change. Issue 1 (Windows/MSVC C4099, GCC/Clang -Wmismatched-tags): the previous 'class Node' -> 'struct Node' change in matmul_nbits_workspace_estimate.h only relocated the tag mismatch. The header has exactly two includers living in different Node 'worlds': cuda_execution_provider.cc (provider_api.h: 'struct Node') and matmul_nbits_e2e_workspace_test.cc (core/graph/graph.h: 'class Node'). Any single tag we pick mismatches one of them. Both includers already declare Node (via their own core/bridge headers) BEFORE including this header, so we now omit the forward declaration entirely and rely on the includer's declaration, with a comment explaining why. Verified include order: provider_api.h precedes the header in the EP TU; core/graph/graph.h precedes it in the test TU. Issue 2 (portability): the pre-existing EndToEndWorkspaceAgreement (Test A/B) only checked cudaGetDeviceCount() > 0, so on a real GPU with compute capability < 7.5 its ASSERT_TRUE(level1.has_value()) would FAIL rather than skip (production CheckFpAIntBEligibility rejects fpA_intB below SM 7.5). Applied the same CudaDeviceComputeCapabilityOrNegative() + kMinFpAIntBSm >= 75 GTEST_SKIP guard already used by Tests C/D. Minor cleanups: extracted FindNodeByOpType(graph, op_type) helper to replace the four duplicated node-search loops, and documented why the Add control test (Test E) needs only a device-presence check (no SM>=7.5 requirement).

Copilot

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 3 comments.

…nit-test TU with DISABLE_CONTRIB_OPS - matmul_nbits_e2e_workspace_test.cc: removed accidentally duplicated header comment block (lines were repeated verbatim). - matmul_nbits_workspace_test.cc: this TU is globbed into onnxruntime_providers_cuda_ut unconditionally, but includes matmul_nbits.h (a contrib op header) and references MatMulNBits<T> symbols that don't exist when DISABLE_CONTRIB_OPS is set, since contrib_ops sources are excluded from the build in that configuration -> would cause link failures. Guard the file the same way as its companion e2e test (!defined(DISABLE_CONTRIB_OPS) && USE_FPA_INTB_GEMM), and update the trailing #endif comment to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

… CI check Unrelated to this PR's MatMulNBits workspace-estimation work; the branch was 23 commits behind main and docs/ContribOperators.md / docs/OperatorKernels.md had drifted out of sync with main's QMoE nvfp4 support, causing the 'Windows GPU Kernel Documentation Validation' CI check to fail.

Tianlei Wu (tianleiwu)

Re-reviewed at head 5e4d471. The two-level workspace-estimation pilot is in good shape and my earlier blocking concern is resolved.

Verified this round:

Single-source-of-truth refactor is behavior-preserving. CheckFpAIntBEligibility reproduces the old constructor gate: the new fp16/bf16 dtype gate is a no-op inside the if constexpr (MLFloat16||BFloat16) block, and the new weight_prepacked/SM90 validations are unreachable in the constructor because ORT_ENFORCE(weight_prepacked_ in {0,1,2}) and ValidateSm90PrepackedWeightSupport(sm_, block_size_) already throw earlier for those cases. kInputIndexGroupIndex == 4 matches on both paths.
Level-2 vs runtime arch consistency is correct. getWorkspaceSize reads the runner's sm_, which setArch(FpAIntBPackingSmForKernel()) already overwrote to the effective arch during InitGemmProfiler, so it agrees with the Level-2 effective_sm even on an SM90 device running the SM80-compat layout.
Overflow handling is sound. ORT's SafeInt uses SafeIntExceptionHandler<OnnxRuntimeException> (ORT_THROW), so the catch (const OnnxRuntimeException&) in ComputeFpAIntBGemmWorkspaceSize / EstimateMatMulNBitsWorkspace / DeclareWorkspaceRequirements correctly catches overflow and narrowing failures and degrades to nullopt/dynamic-fallback. The new SM90 SafeInt<size_t> path also fixes a latent pre-cast int overflow in the old formula.
Cmake fix confirmed: the workspace tests are now excluded from the plugin-internal list with an explanatory comment; resolving that thread.

Dismiss request change since there is no blocking issue in latest commit.

…ignment field - last_compute_workspace_bytes_ (test-only instrumentation) now uses memory_order_relaxed for both store and load, since the pilot's tests are single-threaded and don't need seq_cst ordering. - Add an optional std::optional<size_t> alignment field to WorkspaceRequirement, defaulting to nullopt. No kernel sets it today (CUDA's allocator already guarantees >= 256-byte alignment, sufficient for MatMulNBits' and GroupQueryAttention's/Attention's workspace buffers), but it reserves the field now to avoid an ABI-breaking change later if a future kernel needs stricter alignment or a shared-arena packer needs to co-locate multiple kernels' slots. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ade1ab12-1137-46bb-a92b-e730a6f28bf4

- cuda_kernel_workspace_inventory.md: reclassify MatMulNBits/fpA_intB_GEMM static determinability from upper-bound-only to exact, confirmed by this pilot's closed-form workspace formula (m, n, k, sm, multiProcessorCount). MOE is left unchanged (different runner, not investigated here). - future_directions_constrained_env.md: sync the WorkspaceRequirement struct with the new alignment field; document that neither Level 1 nor Level 2 yet receives real runtime shapes (only this pilot's test harness calls them today); document the confirmed math-helper vs. graph-parsing reuse boundary; enumerate the remaining concrete cost of a real plugin-side override (answers reviewer question about the plugin EP plan). - New skill workspace-estimation-shared-header: captures the Node forward-declaration cross-DLL-world pitfall and the shared-header graph-type-free rule for future agents working on this boundary. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ade1ab12-1137-46bb-a92b-e730a6f28bf4

…lity - WorkspaceRequirement.alignment: std::optional<size_t> -> size_t alignment_bytes (0 = allocator default). critical-reviewer flagged that std::optional's layout is not guaranteed stable across compilers/STL versions, which undermines the struct's own stated goal of being usable across a future plugin-DLL boundary. A scalar with a sentinel value avoids that risk. Updated both construction sites and rebuilt/retested (CUDA_EP_Unittest.All: 58 pass / 2 unrelated skip / 0 fail). - cuda_kernel_workspace_inventory.md / future_directions_constrained_env.md / workspace-estimation-shared-header skill: applied readability-reviewer's findings — moved long table-cell caveats to footnotes, shortened the alignment_bytes inline code comment (moved rationale to prose), removed a stray double blank line, de-duplicated a takeaway bullet, added a matching "how to confirm" step to the skill's second pitfall, and added MatMulNBits/fpA_intB to the multiProcessorCount row in the API-requirements table (previously missing despite being covered in the text above). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ade1ab12-1137-46bb-a92b-e730a6f28bf4

Initial plan

2a23f93

Copilot AI assigned Copilot and Ti-Tai Wang (titaiwangms) Jul 21, 2026

Copilot started work on behalf of Ti-Tai Wang (titaiwangms) July 21, 2026 20:09 View session

Copilot AI linked an issue Jul 21, 2026 that may be closed by this pull request

[Pilot] Two-level (partition-time + kernel-instance) workspace estimation for MatMulNBits (CUDA EP, fpA_intB/CUTLASS) #29810

Closed

Add two-level workspace estimation for MatMulNBits (CUDA fpA_intB)

4e4c45d

Copilot AI changed the title ~~[WIP] Implement two-level workspace estimation for MatMulNBits~~ Two-level workspace estimation for MatMulNBits (CUDA EP, fpA_intB/CUTLASS) Jul 21, 2026

Copilot finished work on behalf of Ti-Tai Wang (titaiwangms) July 21, 2026 20:32

Copilot AI requested a review from Ti-Tai Wang (titaiwangms) July 21, 2026 20:32

Ti-Tai Wang (titaiwangms) requested a review from Copilot July 21, 2026 20:33

Copilot started reviewing on behalf of Ti-Tai Wang (titaiwangms) July 21, 2026 20:33 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread onnxruntime/core/providers/cuda/cuda_execution_provider.cc Outdated

Comment thread onnxruntime/core/providers/cuda/cuda_execution_provider.cc Outdated

Comment thread onnxruntime/contrib_ops/cuda/quantization/matmul_nbits_workspace_estimate.h Outdated

Copilot AI added 2 commits July 21, 2026 22:10

Tianlei Wu (tianleiwu) previously requested changes Jul 21, 2026

View reviewed changes

Comment thread cmake/onnxruntime_unittests.cmake Outdated

Copilot AI and others added 4 commits July 23, 2026 17:14

Ti-Tai Wang (titaiwangms) requested a review from Copilot July 23, 2026 20:59

Copilot started reviewing on behalf of Ti-Tai Wang (titaiwangms) July 23, 2026 21:00 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

Comment thread onnxruntime/test/providers/cuda/test_cases/matmul_nbits_e2e_workspace_test.cc Outdated

Comment thread onnxruntime/test/providers/cuda/test_cases/matmul_nbits_workspace_test.cc Outdated

Comment thread onnxruntime/test/providers/cuda/test_cases/matmul_nbits_workspace_test.cc Outdated

Copilot AI and others added 2 commits July 23, 2026 21:14

Ti-Tai Wang (titaiwangms) marked this pull request as ready for review July 23, 2026 21:22

Tianlei Wu (tianleiwu) reviewed Jul 23, 2026

View reviewed changes

Comment thread onnxruntime/contrib_ops/cuda/quantization/matmul_nbits.cc Outdated

Tianlei Wu (tianleiwu) reviewed Jul 23, 2026

View reviewed changes

Comment thread include/onnxruntime/core/framework/workspace_requirement.h

Tianlei Wu (tianleiwu) reviewed Jul 23, 2026

View reviewed changes

Comment thread onnxruntime/contrib_ops/cuda/quantization/matmul_nbits.h

Copilot AI and others added 3 commits July 24, 2026 22:09

Ti-Tai Wang (titaiwangms) requested a review from Tianlei Wu (tianleiwu) July 24, 2026 23:35

Tianlei Wu (tianleiwu) approved these changes Jul 25, 2026

View reviewed changes

Ti-Tai Wang (titaiwangms) merged commit 5a7215b into main Jul 27, 2026
90 checks passed

Ti-Tai Wang (titaiwangms) deleted the copilot/pilot-two-level-workspace-estimation branch July 27, 2026 16:10

Ti-Tai Wang (titaiwangms) mentioned this pull request Aug 5, 2026

Wire max-shape inference into workspace estimation #31613

Merged

BrewTestBot mentioned this pull request Aug 12, 2026

onnxruntime 1.29.0 Homebrew/homebrew-core#298388

Merged

Ti-Tai Wang (titaiwangms) mentioned this pull request Aug 31, 2026

Register Level-1 workspace estimators with kernel metadata #32341

Open

Navigation Menu

Two-level workspace estimation for MatMulNBits (CUDA EP, fpA_intB/CUTLASS) - #29811

Two-level workspace estimation for MatMulNBits (CUDA EP, fpA_intB/CUTLASS)#29811
Ti-Tai Wang (titaiwangms) merged 13 commits into
mainfrom
copilot/pilot-two-level-workspace-estimation

Copilot AI commented Jul 21, 2026 •

edited

Loading

Uh oh!

azure-pipelines Bot commented Jul 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Ti-Tai Wang (titaiwangms) commented Jul 21, 2026

Uh oh!

Tianlei Wu (tianleiwu) left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Tianlei Wu (tianleiwu) left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Copilot AI commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Motivation and Context

Uh oh!

azure-pipelines Bot commented Jul 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Ti-Tai Wang (titaiwangms) commented Jul 21, 2026

Major

Minor

Uh oh!

Tianlei Wu (tianleiwu) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Tianlei Wu (tianleiwu) left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Copilot AI commented Jul 21, 2026 •

edited

Loading

Tianlei Wu (tianleiwu) left a comment •

edited

Loading