Uh oh!
There was an error while loading. Please reload this page.
fix: build the reference runner with the backend that registers the CUDA allocator - #4536
Closed
shoumikhin wants to merge 1 commit into
Closed
fix: build the reference runner with the backend that registers the CUDA allocator#4536shoumikhin wants to merge 1 commit into
shoumikhin wants to merge 1 commit into
Conversation
…UDA allocator The reference runner cannot run the programs this project exports. It loads one, initializes the TensorRT engine, then stops on the very first instruction: op__device_copy.cpp] Check failed (allocator != nullptr): _h2d_copy: no device allocator registered for device_type=1 method.cpp] KernelCall failed at instruction 0:0 in et_copy::_h2d_copy.out: 0x20 main.cpp] execute() failed on run 0: 0x20 A Torch-TensorRT export marks its delegate inputs and outputs as CUDA memory, so ExecuTorch's PropagateDevicePass places et_copy::_h2d_copy and _d2h_copy around the delegate. Those kernels ask the runtime for the allocator registered for the CUDA device type. In ExecuTorch 1.4.1 the only thing that registers one is the CUDA backend, from a static initializer in backends/cuda/runtime/cuda_backend.cpp, and the runner was built without it. So the option is not optional here, even for a program that uses only the TensorRT delegate. The copies are on the default export path, the one where the caller passes host tensors, which is exactly what this runner does. Linking is already handled: the runner links executorch::backends, which includes aoti_cuda_backend once the option is on, and backends/cuda/CMakeLists.txt calls executorch_target_link_options_shared_lib on it, so the static initializer is not dropped. That matters, because a static initializer in a plain static library is dropped when nothing references it. Measured on the release toolchain with the flags this build uses: plain g++ main.o libbackend.a initializer dropped gold, --gc-sections initializer dropped gold, --gc-sections, --whole-archive initializer ran EXECUTORCH_BUILD_EXTENSION_TENSOR is the companion ExecuTorch's own preset checker requires alongside EXECUTORCH_BUILD_CUDA. This was invisible until now. The ExecuTorch runtime wheel build had not completed since 2026-08-14, so the job that runs this check never reached the runner. Note for reviewers: the CUDA toolkit is now required to build the example, and the build is larger. It does not add a libtorch dependency, which the verification script already asserts.
shoumikhin
commented
Aug 20, 2026
ContributorAuthor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The reference runner cannot run the programs this project exports. It loads one, initializes the TensorRT engine, then stops on the very first instruction:
Why
A Torch-TensorRT export marks its delegate inputs and outputs as CUDA memory, so ExecuTorch's
PropagateDevicePassplaceset_copy::_h2d_copyand_d2h_copyaround the delegate. Those kernels ask the runtime for the allocator registered for the CUDA device type.In ExecuTorch 1.4.1, the version pinned here, exactly one thing registers that allocator, a static initializer in
backends/cuda/runtime/cuda_backend.cpp:The runner was built without that backend. So the option is not optional here, even for a program that uses only the TensorRT delegate. These copies are on the default export path, the one where the caller passes host tensors, which is exactly what this runner does.
Registering our own allocator instead is not an option:
DeviceAllocatorRegistry::register_allocatoraborts on a second registration for the same device type, which would break any process that also loads ExecuTorch's CUDA backend.The change
Two options in the example's CMake. Linking needs no change: the runner already links
executorch::backends, which includesaoti_cuda_backendonce the option is on, andbackends/cuda/CMakeLists.txtcallsexecutorch_target_link_options_shared_libon it so the static initializer survives.That last part matters. A static initializer in a plain static library is dropped when nothing references it. Measured on the release toolchain with the flags this build uses:
g++ main.o libbackend.agold,--gc-sectionsgold,--gc-sections,--whole-archiveExecuTorch already applies the equivalent, so nothing extra is needed here.
Why this was invisible
The ExecuTorch runtime wheel build had not completed since 2026-08-14, so the job that builds and runs this runner never got that far. #4534 fixes the build; this is the next thing behind it.
For reviewers
The CUDA toolkit is now required to build the example, and the build is larger. It does not add a libtorch dependency, which
verify-executorch-reference-runner.shalready asserts and would catch.Testing
The link behaviour above was measured in the release container image (
manylinux2_28-builder, gcc 13.3.1). The end to end result comes from CI on this PR, which is the first time this path can run at all.