Uh oh!
There was an error while loading. Please reload this page.
fix: guard the 3D convolutions the pad-folding pass hides from the conv guard - #4644
Open
tp5uiuc wants to merge 1 commit into
Open
fix: guard the 3D convolutions the pad-folding pass hides from the conv guard#4644tp5uiuc wants to merge 1 commit into
tp5uiuc wants to merge 1 commit into
Conversation
This was referenced Aug 29, 2026
tp5uiucforce-pushed
the
tp5uiuc/trtrtx-turing-conv3d-padfold
branch
from
August 29, 2026 19:05
1262c9d to
a8de90fComparetp5uiucforce-pushed
the
tp5uiuc/trtrtx-turing-conv3d-padfold
branch
from
August 29, 2026 23:01
a8de90f to
3dd0c1bComparetp5uiuc
commented
Aug 30, 2026
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
tp5uiucforce-pushed
the
tp5uiuc/trtrtx-turing-conv3d-padfold
branch
from
August 30, 2026 00:49
3dd0c1b to
749d1b9Compare…nv guard convolution_capability_validator rejects forward 3D convolutions when Turing (SM 7.5) is a build target, because TensorRT-RTX builds an engine for them whose createExecutionContext() then returns nullptr. It is registered on aten.convolution.default only. The fuse_pad_into_convolution lowering pass rewrites constant_pad_nd -> convolution into tensorrt::conv_asym_pad, and that op's converter had no capability_validator at all. By the time partitioning runs there is no aten.convolution node left to validate, so a padded 3D convolution reaches TensorRT-RTX regardless of the guard and surfaces at runtime as Expected exec_ctx_.get() != nullptr to be true but got false Unable to (re)create TensorRT execution context This is a production gap, not just a test gap. The pass runs in the normal compile path, and it fires for any non-transposed, zero-fill, non-negative constant_pad_nd -> conv pair -- asymmetry is not required -- so any real model with a padded conv3d hits it on Turing. * aten_ops_converters.py: extract turing_rejects_forward_convolution() so the "3D forward conv is unsupported on this target" rule lives in one place. convolution_capability_validator delegates to it; its behaviour is unchanged, including failing open when meta["val"] is absent. * custom_ops_converters.py: add conv_asym_pad_capability_validator and register it on the fused op. It reads the spatial rank off the argument list rather than node.meta -- the fused op's args are (source, weight, bias, stride, pre_padding, post_padding, dilation, groups), so len(stride) is the rank -- which makes the guard hold whether or not the caller ran the dynamo tracer. The pass never fuses a transposed convolution, so there is no deconvolution case to handle here. Only rank 3 is rejected; 2D keeps running on TensorRT. * test_fuse_pad_into_convolution.py: skip test_padded_conv3d on Turing. DispatchTestCase has no PyTorch-fallback path -- run_test hands the graph straight to TRTInterpreter, skipping the partitioner -- so a rejected node raises UnsupportedOperatorException instead of falling back. Guarding without skipping would only swap one failure for another. * test_turing_capability_guards.py: add a padded-conv3d fallback case and a padded-conv2d positive control. Both key on target_compute_capabilities=[(7, 5)] rather than the live device, so they exercise the guard on any GPU and therefore in CI. Reverting just the registration makes the conv3d case fail, so it covers the gap rather than merely restating it. Deliberately not fixed in the lowering pass. Declining to fuse when the underlying convolution would be rejected keeps one rule in one place and would pick up any future guard on aten.convolution for free, and the pass already receives the settings it would need. But TestFusePadIntoConvolutionPass builds its graphs with a default CompilationSettings(), and test_graph_contains_fused_op_after_lowering calls post_lowering with one too. Default settings mean target_compute_capabilities=None, which resolves against the current device -- so on a Turing GPU the pass would decline to fuse and the five tests that assert the fusion *does* happen would fail. Each would then need a pinned non-Turing target or a skip. Testing (T4 / SM 7.5 and L40S / SM 8.9, driver 595.58.03, identical pinned stacks): * All 8 test_padded_conv3d cases go fail -> skip on Turing and still run and pass on the L40S control, confirming the guard is inert off Turing. * Full lowering/ sweep on both arms, all 259 tests accounted for on each. On Turing exactly 8 tests change status, all of them fail -> skip and all of them test_padded_conv3d: 9 -> 1 failed, 4 -> 12 skipped, and 242 passing either way. (The other 4 pre-existing skips, and 4 fp16 SDPA tests that abort the worker, are unrelated and unchanged.) On the non-Turing control every one of the 259 keeps its previous status and the skip count stays at 0. * The 11 other tests in that file stay green on both arms -- in particular test_graph_contains_fused_op_after_lowering and test_fuses_causal_3d_pad, both of which build 3D graphs and assert the fusion still happens on the T4. * test_turing_capability_guards.py passes on both arms: T4 17 -> 19 passed / 1 skipped, L40S 12 -> 14 passed / 6 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
tp5uiucforce-pushed
the
tp5uiuc/trtrtx-turing-conv3d-padfold
branch
from
August 30, 2026 00:52
749d1b9 to
3c19297Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What — Registers a capability validator on the
tensorrt::conv_asym_padcustom op so a padded 3Dconvolution falls back to PyTorch on Turing.
Why — The
fuse_pad_into_convolutionlowering pass rewritesconstant_pad_nd -> convolutioninto
tensorrt::conv_asym_pad, erasing theaten.convolutionnode the 3D-conv guard keys on.That converter was registered with no
capability_validator, so a padded conv3d walked straight pastthe guard and hit a null execution context (
TRTEngine.cpp:959). It fires for any non-transposed,zero-fill, non-negative pad, so symmetric cases fail too. This is a production path, not just tests.
How — Share the convolution capability predicate between the aten converter and the fused
custom op, reading rank from the fused node, so the guard survives the rewrite instead of being
erased with the node it keyed on.
Testing — Confirmation sweep, both arms: 8
lowering/failures closed, 0 Turing-specificremaining in that module. No status change on the L40S.
Cost / Gotchas — Accounts for 8 of the 105 tests the guards switch off on Turing.
🤖 Generated with Claude Code