Skip to content

Fix GEMM profiler verification across all batches - #3558

Open
meiyou666 wants to merge 1 commit into
NVIDIA:mainfrom
meiyou666:fix/gemm-profiler-batch-verification
Open

Fix GEMM profiler verification across all batches#3558
meiyou666 wants to merge 1 commit into
NVIDIA:mainfrom
meiyou666:fix/gemm-profiler-batch-verification

Conversation

@meiyou666

Copy link
Copy Markdown

Summary

  • Compare each logical GEMM batch independently through batch_data() and batch_stride(), while preserving the existing bitwise-exact and epsilon-based relative comparison semantics.
  • Apply batched comparison to standard GEMM host, device, and cuBLAS verification; block-scaled GEMM D and SFD outputs; and blockwise GEMM D output.
  • Set CUBLASLT_MATRIX_LAYOUT_BATCH_COUNT and CUBLASLT_MATRIX_LAYOUT_STRIDED_BATCH_OFFSET on the cuBLASLt A/B/C/D layouts for batched GEMM.
  • Limit the generic reference loop to host and device reference providers so dedicated provider results remain intact.

Validation

  • Completed full profiler builds on an RTX 4090 with CUDA 11.8 using both CUTLASS_ENABLE_CUBLAS=OFF and CUTLASS_ENABLE_CUBLAS=ON.
  • Ran 38 total host-reference matrix cases across the baseline and fixed profilers at beta=0, using batch_count values 1, 2, 3, 7, and 16. Clean cases passed. On the baseline, batch-0 faults were detected while nonzero-batch faults reproduced the false pass. With the fix, every injected fault was detected.
  • Ran 19 cuBLAS matrix cases with the fixed profiler at beta=0 across the same batch counts and fault positions. Clean cases passed, and all batch-0 and nonzero-batch faults were detected.
  • The full builds compiled the block-scaled and blockwise paths; those paths were not run on this GPU.

Fixes#3511.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GEMM profiler verification compares only batch 0 when batch-count > 1

1 participant

@meiyou666