Uh oh!
There was an error while loading. Please reload this page.
fix: fall back to PyTorch for ops TensorRT-RTX cannot serve on Turing (SM 7.5) - #4546
Closed
tp5uiuc wants to merge 1 commit into
Closed
fix: fall back to PyTorch for ops TensorRT-RTX cannot serve on Turing (SM 7.5)#4546tp5uiuc wants to merge 1 commit into
tp5uiuc wants to merge 1 commit into
Conversation
TensorRT-RTX supports SM 7.5 and up, but its support matrix states that on Turing it does not support FP32 GEMMs or 3D convolutions, and Turing has no bfloat16 hardware at all. Torch-TensorRT had no notion of this and handed those ops to TensorRT-RTX anyway. On a Turing GPU that produces: - FP32 GEMM, static shapes: createExecutionContext() returns null - FP32 GEMM, dynamic shapes: the engine builds and runs, returning an all-zero tensor of the correct shape and dtype, with no exception - 3D convolution: null execution context - bfloat16: segmentation fault The dynamic-shape GEMM case is the motivating one, since it fails silently. Guards key off the compute capabilities being built for rather than the build host, via a new target_compute_capabilities option, so an ahead-of-time build for another architecture partitions correctly instead of baking in the build machine's capabilities. The same list drives setComputeCapability(), keeping partitioning and engine targeting consistent, and is engine-invariant so a cached engine built for different targets is never reused. The convolution guard covers forward 3D convolution only; transposed 3D convolution works on Turing and is left on TensorRT. The GEMM guard keys on fp32 operands, so fp16 GEMMs accumulating in fp32 are unaffected. bfloat16 is gated in the partitioners rather than per-converter because the crash is not operator-specific, mirroring the existing complex-dtype handling. Converter unit tests build graphs with empty node meta, so dtype-based capability validators cannot fire there; the affected tests skip explicitly. Also binds one pre-existing untyped return in _settings.py to a typed local: that file is now in the changed set, so --strict mypy blocks the commit on it.
tp5uiuc
commented
Aug 29, 2026
CollaboratorAuthor
Closing in favour of a reviewable series. This PR was one commit; the work it started grew to Replaced by six PRs. Three are not Turing bugs at all — they reproduce on an L40S with
The Turing work is a stack, based on #4643:
Three things found after this PR was opened are folded into #4643 rather than shipped as
Verified at the series tip on a T4 (SM 7.5) and an L40S (SM 8.9), driver-matched, all six test Not fixed: 5 |
tp5uiuc added a commit
that referenced
this pull request
Aug 29, 2026
impl.normalization.cdist_forward computes p == 2 with a matrix-multiply layer for compute_mode 1, or 0/absent with an operand above the row threshold. That GEMM is emitted inside the converter, so the graph holds a single _cdist_forward node and no mm/bmm for gemm_capability_validator to reject, and on Turing TensorRT-RTX fails cuDNN graph compilation with "No valid engine configs for Matmul_MUL_SUB_SQRT_", leaving a null execution context. Zero tests were failing: PR #4546 skipped the covering tests rather than guarding the converter, so nothing was red while the conversion path stayed open. A skip protects CI; it does not protect a caller. Which arguments emit a GEMM is decided by cdist_emits_matmul, the predicate cdist_forward itself uses, so the guard cannot drift from the branch it mirrors. Unlike gemm_capability_validator this is deliberately not dtype-keyed: measured on a T4 the fused pattern fails for FP16 operands too, so what predicts failure is whether the matmul layer is emitted, not its precision. The row threshold is load-bearing rather than an optimisation -- below it no GEMM is emitted, and rejecting anyway would regress, since PyTorch's cdist_cuda has no Half kernel. One behaviour note: where exactly one operand's row count is statically known and exceeds the threshold, the shared predicate rejects, whereas reading the shapes directly would have failed open. Rejecting is correct -- the converter's condition is an or, so a single known operand above the threshold already decides it. The case is unreachable today because the converter is not registered supports_dynamic_shapes, so the partitioner refuses a partially dynamic cdist first.
tp5uiuc added a commit
that referenced
this pull request
Aug 30, 2026
impl.normalization.cdist_forward computes p == 2 with a matrix-multiply layer for compute_mode 1, or 0/absent with an operand above the row threshold. That GEMM is emitted inside the converter, so the graph holds a single _cdist_forward node and no mm/bmm for gemm_capability_validator to reject, and on Turing TensorRT-RTX fails cuDNN graph compilation with "No valid engine configs for Matmul_MUL_SUB_SQRT_", leaving a null execution context. Zero tests were failing: PR #4546 skipped the covering tests rather than guarding the converter, so nothing was red while the conversion path stayed open. A skip protects CI; it does not protect a caller. Which arguments emit a GEMM is decided by cdist_emits_matmul, the predicate cdist_forward itself uses, so the guard cannot drift from the branch it mirrors. Unlike gemm_capability_validator this is deliberately not dtype-keyed: measured on a T4 the fused pattern fails for FP16 operands too, so what predicts failure is whether the matmul layer is emitted, not its precision. The row threshold is load-bearing rather than an optimisation -- below it no GEMM is emitted, and rejecting anyway would regress, since PyTorch's cdist_cuda has no Half kernel. One behaviour note: where exactly one operand's row count is statically known and exceeds the threshold, the shared predicate rejects, whereas reading the shapes directly would have failed open. Rejecting is correct -- the converter's condition is an or, so a single known operand above the threshold already decides it. The case is unreachable today because the converter is not registered supports_dynamic_shapes, so the partitioner refuses a partially dynamic cdist first.
tp5uiuc added a commit
that referenced
this pull request
Aug 30, 2026
impl.normalization.cdist_forward computes p == 2 with a matrix-multiply layer for compute_mode 1, or 0/absent with an operand above the row threshold. That GEMM is emitted inside the converter, so the graph holds a single _cdist_forward node and no mm/bmm for gemm_capability_validator to reject, and on Turing TensorRT-RTX fails cuDNN graph compilation with "No valid engine configs for Matmul_MUL_SUB_SQRT_", leaving a null execution context. Zero tests were failing: PR #4546 skipped the covering tests rather than guarding the converter, so nothing was red while the conversion path stayed open. A skip protects CI; it does not protect a caller. Which arguments emit a GEMM is decided by cdist_emits_matmul, the predicate cdist_forward itself uses, so the guard cannot drift from the branch it mirrors. Unlike gemm_capability_validator this is deliberately not dtype-keyed: measured on a T4 the fused pattern fails for FP16 operands too, so what predicts failure is whether the matmul layer is emitted, not its precision. The row threshold is load-bearing rather than an optimisation -- below it no GEMM is emitted, and rejecting anyway would regress, since PyTorch's cdist_cuda has no Half kernel. One behaviour note: where exactly one operand's row count is statically known and exceeds the threshold, the shared predicate rejects, whereas reading the shapes directly would have failed open. Rejecting is correct -- the converter's condition is an or, so a single known operand above the threshold already decides it. The case is unreachable today because the converter is not registered supports_dynamic_shapes, so the partitioner refuses a partially dynamic cdist first.
tp5uiuc added a commit
that referenced
this pull request
Aug 30, 2026
impl.normalization.cdist_forward computes p == 2 with a matrix-multiply layer for compute_mode 1, or 0/absent with an operand above the row threshold. That GEMM is emitted inside the converter, so the graph holds a single _cdist_forward node and no mm/bmm for gemm_capability_validator to reject, and on Turing TensorRT-RTX fails cuDNN graph compilation with "No valid engine configs for Matmul_MUL_SUB_SQRT_", leaving a null execution context. Zero tests were failing: PR #4546 skipped the covering tests rather than guarding the converter, so nothing was red while the conversion path stayed open. A skip protects CI; it does not protect a caller. Which arguments emit a GEMM is decided by cdist_emits_matmul, the predicate cdist_forward itself uses, so the guard cannot drift from the branch it mirrors. Unlike gemm_capability_validator this is deliberately not dtype-keyed: measured on a T4 the fused pattern fails for FP16 operands too, so what predicts failure is whether the matmul layer is emitted, not its precision. The row threshold is load-bearing rather than an optimisation -- below it no GEMM is emitted, and rejecting anyway would regress, since PyTorch's cdist_cuda has no Half kernel. One behaviour note: where exactly one operand's row count is statically known and exceeds the threshold, the shared predicate rejects, whereas reading the shapes directly would have failed open. Rejecting is correct -- the converter's condition is an or, so a single known operand above the threshold already decides it. The case is unreachable today because the converter is not registered supports_dynamic_shapes, so the partitioner refuses a partially dynamic cdist first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
TensorRT-RTX supports SM 7.5 and up, but its support matrix states that on Turing it "does not support FP32 GEMMs and 3D convolutions in this release", and Turing has no bfloat16 hardware. Torch-TensorRT had no notion of this and handed those ops to TensorRT-RTX anyway. Observed on a Turing GPU:
createExecutionContext()returns null.How
_utils.py:get_target_compute_capabilities()/trt_rtx_targets_turing(). Guards key off the capabilities being built for, not the build host — querying the local device would bake the build machine into an ahead-of-time artifact.aten_ops_converters.py:gemm_capability_validatoron matmul/mm/bmm/dot/mv/addmm (fp32 operands only, so fp16 withuse_fp32_accis unaffected); 3D convolution rejected inconvolution_capability_validator— forward only, since transposed 3D works on Turing.partitioning/: bfloat16 gated in both partitioners rather than per-converter, as the crash is not operator-specific. Mirrors the existing complex-dtype handling._settings.py/_defaults.py/_compiler.py: newtarget_compute_capabilitiesoption on all three compile entry points, added to_SETTINGS_TO_BE_ENGINE_INVARIANTso a cached engine built for different targets is never reused._TRTInterpreter.py: the same list drivessetComputeCapability(), so partitioning and engine targeting cannot drift apart.skip_if_trt_rtx_turing()in the conversion harness, applied to the affected matmul, cdist (p == 2), convolution (3D) and binary-op (bf16) tests.Testing
On a Turing GPU all four cases now fall back to PyTorch and produce correct results (cosine 1.000000), while fp16 GEMM, transposed 3D convolution and 2D convolution still run on TensorRT. On an SM 8.9 GPU behaviour is unchanged by default, and compiling with
target_compute_capabilities=[(7, 5)]reproduces Turing's partitioning — so this is testable without Turing hardware. Affected conversion suites: 121 passed, 57 skipped, 0 failed (previously 23 failures).Notes for reviewers
meta["val"], as other validators in this module do. Converter unit tests build graphs with empty node meta, so validators cannot fire there at all; those tests skip explicitly instead._settings.pyis bound to a typed local, because that file is now in the changed set and--strictmypy blocks the commit on it.Type of change
Checklist: