Skip to content

fix: build the reference runner with the backend that registers the CUDA allocator - #4536

Closed
shoumikhin wants to merge 1 commit into
pytorch:mainfrom
shoumikhin:fix-reference-runner-cuda-allocator
Closed

fix: build the reference runner with the backend that registers the CUDA allocator#4536
shoumikhin wants to merge 1 commit into
pytorch:mainfrom
shoumikhin:fix-reference-runner-cuda-allocator

Conversation

@shoumikhin

Copy link
Copy Markdown
Contributor

Summary

The reference runner cannot run the programs this project exports. It loads one, initializes the TensorRT engine, then stops on the very first instruction:

op__device_copy.cpp] Check failed (allocator != nullptr):
_h2d_copy: no device allocator registered for device_type=1
method.cpp] KernelCall failed at instruction 0:0 in et_copy::_h2d_copy.out: 0x20
main.cpp] execute() failed on run 0: 0x20

Why

A Torch-TensorRT export marks its delegate inputs and outputs as CUDA memory, so ExecuTorch's PropagateDevicePass places et_copy::_h2d_copy and _d2h_copy around the delegate. Those kernels ask the runtime for the allocator registered for the CUDA device type.

In ExecuTorch 1.4.1, the version pinned here, exactly one thing registers that allocator, a static initializer in backends/cuda/runtime/cuda_backend.cpp:

staticbool cuda_allocator_registered = [] {
executorch::runtime::register_device_allocator(&cuda::CudaAllocator::instance());
returntrue;
}();

The runner was built without that backend. So the option is not optional here, even for a program that uses only the TensorRT delegate. These copies are on the default export path, the one where the caller passes host tensors, which is exactly what this runner does.

Registering our own allocator instead is not an option: DeviceAllocatorRegistry::register_allocator aborts on a second registration for the same device type, which would break any process that also loads ExecuTorch's CUDA backend.

The change

Two options in the example's CMake. Linking needs no change: the runner already links executorch::backends, which includes aoti_cuda_backend once the option is on, and backends/cuda/CMakeLists.txt calls executorch_target_link_options_shared_lib on it so the static initializer survives.

That last part matters. A static initializer in a plain static library is dropped when nothing references it. Measured on the release toolchain with the flags this build uses:

linkresult
g++ main.o libbackend.ainitializer dropped
gold, --gc-sectionsinitializer dropped
gold, --gc-sections, --whole-archiveinitializer ran

ExecuTorch already applies the equivalent, so nothing extra is needed here.

Why this was invisible

The ExecuTorch runtime wheel build had not completed since 2026-08-14, so the job that builds and runs this runner never got that far. #4534 fixes the build; this is the next thing behind it.

For reviewers

The CUDA toolkit is now required to build the example, and the build is larger. It does not add a libtorch dependency, which verify-executorch-reference-runner.sh already asserts and would catch.

Testing

The link behaviour above was measured in the release container image (manylinux2_28-builder, gcc 13.3.1). The end to end result comes from CI on this PR, which is the first time this path can run at all.

…UDA allocator
The reference runner cannot run the programs this project exports. It loads one,
initializes the TensorRT engine, then stops on the very first instruction:
op__device_copy.cpp] Check failed (allocator != nullptr):
_h2d_copy: no device allocator registered for device_type=1
method.cpp] KernelCall failed at instruction 0:0 in et_copy::_h2d_copy.out: 0x20
main.cpp] execute() failed on run 0: 0x20
A Torch-TensorRT export marks its delegate inputs and outputs as CUDA memory, so
ExecuTorch's PropagateDevicePass places et_copy::_h2d_copy and _d2h_copy around
the delegate. Those kernels ask the runtime for the allocator registered for the
CUDA device type. In ExecuTorch 1.4.1 the only thing that registers one is the
CUDA backend, from a static initializer in backends/cuda/runtime/cuda_backend.cpp,
and the runner was built without it.
So the option is not optional here, even for a program that uses only the TensorRT
delegate. The copies are on the default export path, the one where the caller
passes host tensors, which is exactly what this runner does.
Linking is already handled: the runner links executorch::backends, which includes
aoti_cuda_backend once the option is on, and backends/cuda/CMakeLists.txt calls
executorch_target_link_options_shared_lib on it, so the static initializer is not
dropped. That matters, because a static initializer in a plain static library is
dropped when nothing references it. Measured on the release toolchain with the
flags this build uses:
plain g++ main.o libbackend.a initializer dropped
gold, --gc-sections initializer dropped
gold, --gc-sections, --whole-archive initializer ran
EXECUTORCH_BUILD_EXTENSION_TENSOR is the companion ExecuTorch's own preset checker
requires alongside EXECUTORCH_BUILD_CUDA.
This was invisible until now. The ExecuTorch runtime wheel build had not completed
since 2026-08-14, so the job that runs this check never reached the runner.
Note for reviewers: the CUDA toolkit is now required to build the example, and the
build is larger. It does not add a libtorch dependency, which the verification
script already asserts.
@shoumikhin

Copy link
Copy Markdown
ContributorAuthor

Folded into #4534. This cannot be verified on its own: until the wheel build there succeeds, the job that builds and runs the reference runner never starts, so a standalone PR could not be exercised at all. It is the last commit on #4534 now.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@shoumikhin