Uh oh!
There was an error while loading. Please reload this page.
fix(executorch): copy the engine in bulk instead of byte by byte - #4473
Merged
lanluo-nvidia merged 1 commit intoAug 18, 2026
Merged
Conversation
shoumikhinforce-pushed
the
trt-engine-serialization-perf
branch
from
August 9, 2026 23:37
21ac6db to
4063483Compare
Serializing an engine went through bytes() on the tensors storage object.
shoumikhinforce-pushed
the
trt-engine-serialization-perf
branch
from
August 13, 2026 02:50
4063483 to
e0cf2b9Compare7 tasks
Conarnar added a commit
to Conarnar/TensorRT
that referenced
this pull request
Aug 18, 2026
Export resolves engine info for two purposes, and neither wants the base64 form serialize() produces. validate_engine_program reads only flags; replace_execute_engine wants the engine as a byte tensor. Both were paying an encode in C++ and, for the second, a matching decode here. Take each half from the accessor that provides it: metadata_only=True on get_engine_info_from_state and _resolve_engine_info, so a metadata reader never triggers engine serialization. _resolve_engine_tensor returns the engine as a uint8 tensor directly, which also drops the torch.frombuffer rebuild. Both fall back to the old path when the runtime lacks the accessors, so this does not require a matching runtime. They are missing only when the Torch-TensorRT C++ library is older than this Python package -- a source or editable build where only the Python half was rebuilt. The symptom would otherwise be silence, since the fallback is correct and merely slower, so it warns once per accessor naming the cause. _resolve_engine_object is factored out of _resolve_engine_info because the engine arg is a get_attr before ExecuTorch lifts constants and a placeholder after; handling only the first made the tensor accessor silently fall back to base64 on every graph that had been through staging, which is exactly the case that matters. Nothing is cached. With the accessors present a metadata read is a member read away from free, so there is nothing worth memoizing; a runtime without them re-serializes per read, which is the price of keeping this change small. backend.py and partitioner.py are deliberately untouched. Their engine reads go through the no_op_placeholder branch, which already carries the engine as a tensor argument and serializes nothing, so routing them through the accessors would add a parameter that changes no work. Measured on a base carrying pytorch#4473, with pytorch#4489 present in the loaded runtime, exporting a model with a 67MB engine; six runs per configuration, interleaved. The passes that read engine info go from 1.40s to 0.11s and the ExecuTorch lowering phase from 2.35s to 1.08s, a 54% reduction. Engine-state reads are nearly all of that: 1.07s to 0.07s, with the Python-side base64 decode going from 0.26s to nothing. Serializing the engine is only about 0.07s of a 0.54s __getstate__, so most of what disappears is the base64 encode and the string copies feeding it. A metadata-only read costs 10-20us.
Uh oh!
There was an error while loading. Please reload this page.
Conarnar added a commit
to Conarnar/TensorRT
that referenced
this pull request
Aug 18, 2026
Export resolves engine info for two purposes, and neither wants the base64 form serialize() produces. validate_engine_program reads only flags; replace_execute_engine wants the engine as a byte tensor. Both were paying an encode in C++ and, for the second, a matching decode here. Take each half from the accessor that provides it: metadata_only=True on get_engine_info_from_state and _resolve_engine_info, so a metadata reader never triggers engine serialization. _resolve_engine_tensor returns the engine as a uint8 tensor directly, which also drops the torch.frombuffer rebuild. Both fall back to the old path when the runtime lacks the accessors, so this does not require a matching runtime. They are missing only when the Torch-TensorRT C++ library is older than this Python package -- a source or editable build where only the Python half was rebuilt. The symptom would otherwise be silence, since the fallback is correct and merely slower, so it warns once per accessor naming the cause. _resolve_engine_object is factored out of _resolve_engine_info because the engine arg is a get_attr before ExecuTorch lifts constants and a placeholder after; handling only the first made the tensor accessor silently fall back to base64 on every graph that had been through staging, which is exactly the case that matters. Nothing is cached. With the accessors present a metadata read is a member read away from free, so there is nothing worth memoizing; a runtime without them re-serializes per read, which is the price of keeping this change small. backend.py and partitioner.py are deliberately untouched. Their engine reads go through the no_op_placeholder branch, which already carries the engine as a tensor argument and serializes nothing, so routing them through the accessors would add a parameter that changes no work. Measured on a base carrying pytorch#4473, with pytorch#4489 present in the loaded runtime, exporting a model with a 67MB engine; six runs per configuration, interleaved. The passes that read engine info go from 1.40s to 0.11s and the ExecuTorch lowering phase from 2.35s to 1.08s, a 54% reduction. Engine-state reads are nearly all of that: 1.07s to 0.07s, with the Python-side base64 decode going from 0.26s to nothing. Serializing the engine is only about 0.07s of a 0.54s __getstate__, so most of what disappears is the base64 encode and the string copies feeding it. A metadata-only read costs 10-20us.
Conarnar added a commit
to Conarnar/TensorRT
that referenced
this pull request
Aug 19, 2026
Export resolves engine info for two purposes, and neither wants the base64 form serialize() produces. validate_engine_program reads only flags; replace_execute_engine wants the engine as a byte tensor. Both were paying an encode in C++ and, for the second, a matching decode here. Take each half from the accessor that provides it: metadata_only=True on get_engine_info_from_state and _resolve_engine_info, so a metadata reader never triggers engine serialization. _resolve_engine_tensor returns the engine as a uint8 tensor directly, which also drops the torch.frombuffer rebuild. Both fall back to the old path when the runtime lacks the accessors, so this does not require a matching runtime. They are missing only when the Torch-TensorRT C++ library is older than this Python package -- a source or editable build where only the Python half was rebuilt. The symptom would otherwise be silence, since the fallback is correct and merely slower, so it warns once per accessor naming the cause. _resolve_engine_object is factored out of _resolve_engine_info because the engine arg is a get_attr before ExecuTorch lifts constants and a placeholder after; handling only the first made the tensor accessor silently fall back to base64 on every graph that had been through staging, which is exactly the case that matters. Nothing is cached. With the accessors present a metadata read is a member read away from free, so there is nothing worth memoizing; a runtime without them re-serializes per read, which is the price of keeping this change small. backend.py and partitioner.py are deliberately untouched. Their engine reads go through the no_op_placeholder branch, which already carries the engine as a tensor argument and serializes nothing, so routing them through the accessors would add a parameter that changes no work. Measured on a base carrying pytorch#4473, with pytorch#4489 present in the loaded runtime, exporting a model with a 67MB engine; six runs per configuration, interleaved. The passes that read engine info go from 1.40s to 0.11s and the ExecuTorch lowering phase from 2.35s to 1.08s, a 54% reduction. Engine-state reads are nearly all of that: 1.07s to 0.07s, with the Python-side base64 decode going from 0.26s to nothing. Serializing the engine is only about 0.07s of a 0.54s __getstate__, so most of what disappears is the base64 encode and the string copies feeding it. A metadata-only read costs 10-20us.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
Serializing a TensorRT engine to bytes goes through
bytes(tensor.untyped_storage()).That reads like a bulk copy, but it iterates the storage one element at a time in Python,
at roughly two seconds per megabyte:
For a model whose graph produces many engines, this step dominates the entire export. A
program with 8.8 GB of engines spends about five hours here. One with 70 GB spends closer
to two days. It is easy to mistake for a slow engine build, because the export simply sits
there making no visible progress.
It is also wrong for a view
contiguous()is a no-op for a tensor that is already contiguous, including a row slice ofa larger tensor. In that case the storage holds neighbouring bytes, so the serialized
engine comes out longer than the tensor it came from:
Today's engines happen to own their whole storage, so this does not bite in practice yet.
It is a trap for anyone who later produces the engine buffer as a slice.
The fix
Copy through a
memoryviewover the tensor's own buffer:The comment on the previous code explains what it was avoiding:
.numpy().tobytes()allocates a second full-size buffer, which roughly doubles peak memory for a multi-gigabyte
engine.
memoryviewkeeps that property, since it is a view rather than a copy, whiledoing the copy in one shot and respecting the tensor's bounds.
Testing
Verified byte-for-byte equality against the previous path for the shapes this code sees:
The third row is the bug above: the old path returned the whole storage rather than the
tensor.
Also exported a multi-method model that produces ten engines totalling 8.8 GB. Before the
change the serialization step ran for over an hour and a half without finishing; after it,
the same step completes in minutes and produces a program that loads and runs.