Cortex-M backend: Add AoT scratch-buffer planning.

[Erik-Lundell]

This is done for conv, depthwise conv, transpose conv, and bmm.

Add scratch tensors to the operator signatures, which are then
assigned exir.memory.alloc. These allocs are automatically memory
planned by ExecuTorch.

Introduce required_cmsis_buffer_sizewhich computes the buffer
size from node properties + the Cortex-M configuration.
The function uses functions registered by target in
backends/cortex_m/passes/scratch_buffer_sizes.py
This is used to set the size of the allocs in ConvertToCortexMPass

Finally, modify the kernels to use the new scratch tensor instead
of allocating temporary memory. Add a new macro CORTEX_M_ENABLE_RUNTIME_CHECKS
to do a safety check that the aot computed buffer size is equal to the
buffer size computed at runtime. Use this when testing.

cc @psiddh@AdrianLundell@digantdesai@rascani@freddan80@per@zingo@oscarandersson8218@mansnils@Sebastian-Larsson@robell

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19636

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 3 Unrelated Failures

As of commit 7ad198a with merge base b73df0b ():

NEW FAILURES - The following jobs have failed:

• trunk / test-models-macos-cpu (llama2, xnnpack-quantization-delegation) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-qnn-optimum-model (fp32, pvt) / linux-job (gh)
  RuntimeError: Command docker exec -t 74145e3c8c2a31a529f88874bf2fc5b5e932ca4d6848ce98c8e9b592c10aa5c8 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• trunk / unittest-release / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Erik-Lundell]

This is a polished version of #16580

[rascani]

exir.passes.make_alloc_node is the canonical helper for emitting memory.alloc nodes. It's used by to_out_variant and to_scratch_op_pass use internally, and it sets meta["val"] and meta["tensor_meta"] on the alloc. The memory planner keys off meta["val"] / meta["tensor_meta"] to build the TensorSpec that drives lifetime analysis and arena placement.

[Erik-Lundell]

Thanks, I'll use that

[rascani]

IIUC, the current flow creates each scratch alloc with a placeholder shape, wires it into the cortex_m op's args, and then walks back through node.args[-(i+1)] in _set_scratch_buffer_size to mutate the alloc's shape once the size is known. This works, but the two-phase flow has a few drawbacks:

• The alloc -> size pairing is positional and implicit, split across two files. Adding a non-scratch trailing arg to any op signature silently mis-pairs.
• The UNINITIALIZED_ALLOC_ARGS sentinel + later mutation requires a reader to follow two hops to understand the alloc's actual size.
• _set_scratch_buffer_size exists only to undo the placeholder.

All the values required_cmsis_nn_buffer_sizes needs are already available locally in get*_replacement. If the size functions took inputs directly (e.g. an explicit ConvBufferSizeInputs dataclass), you could compute sizes before constructing the cortex_m op and emit allocs at their final size on one pass.

[Erik-Lundell]

I see your point, but I would like to keep the current design with the following arguments:

• node already perfectly captures the context needed to compute scratch buffer sizes. Introducing a new dataclass interface creates more boilerplate code with the risk of mismatching args.
• Importantly, node.meta["val"].shape is needed for conv, which requires either executing the node the calculate it (as is done now), or some duplication of logic to compute the shape from its args.
• cortex_m node creation is only done in one place in the pass (L513) , and the initialization happens directly after, so the logic is not too far separated.
• There is a check that the trailing args are allocs, so there is no silent mis-pairing. If alloc sizes were given in the wrong order, the runtime check would catch it.

I have pushed a commit to try to clarify the pattern though.

[digantdesai]

Nice!

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: Erik-Lundell / name: Erik Lundell (23c86c1, 264fc9e, 5f1cf38, 699e731, 7ad198a, 936ed4b, f801779)
• ✅ login: mansnils / name: Måns Nilsson (fae7030)

[Erik-Lundell]

This is done for conv, depthwise conv, transpose conv, and bmm.

Add scratch tensors to the operator signatures, which are then
assigned exir.memory.alloc. These allocs are automatically memory
planned by ExecuTorch.

Introduce required_cmsis_buffer_sizewhich computes the buffer
size from node properties + the Cortex-M configuration.
The function uses functions registered by target in
backends/cortex_m/passes/scratch_buffer_sizes.py
This is used to set the size of the allocs in ConvertToCortexMPass

Finally, modify the kernels to use the new scratch tensor instead
of allocating temporary memory. Add a new macro CORTEX_M_ENABLE_RUNTIME_CHECKS
to do a safety check that the aot computed buffer size is equal to the
buffer size computed at runtime. Use this when testing.

cc @psiddh@AdrianLundell@digantdesai@rascani@freddan80@per@zingo@oscarandersson8218@mansnils@Sebastian-Larsson@robell

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19636

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 3 Unrelated Failures

As of commit 7ad198a with merge base b73df0b ():

NEW FAILURES - The following jobs have failed:

• trunk / test-models-macos-cpu (llama2, xnnpack-quantization-delegation) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-qnn-optimum-model (fp32, pvt) / linux-job (gh)
  RuntimeError: Command docker exec -t 74145e3c8c2a31a529f88874bf2fc5b5e932ca4d6848ce98c8e9b592c10aa5c8 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• trunk / unittest-release / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Erik-Lundell]

This is a polished version of #16580

[rascani]

exir.passes.make_alloc_node is the canonical helper for emitting memory.alloc nodes. It's used by to_out_variant and to_scratch_op_pass use internally, and it sets meta["val"] and meta["tensor_meta"] on the alloc. The memory planner keys off meta["val"] / meta["tensor_meta"] to build the TensorSpec that drives lifetime analysis and arena placement.

[Erik-Lundell]

Thanks, I'll use that

[rascani]

IIUC, the current flow creates each scratch alloc with a placeholder shape, wires it into the cortex_m op's args, and then walks back through node.args[-(i+1)] in _set_scratch_buffer_size to mutate the alloc's shape once the size is known. This works, but the two-phase flow has a few drawbacks:

• The alloc -> size pairing is positional and implicit, split across two files. Adding a non-scratch trailing arg to any op signature silently mis-pairs.
• The UNINITIALIZED_ALLOC_ARGS sentinel + later mutation requires a reader to follow two hops to understand the alloc's actual size.
• _set_scratch_buffer_size exists only to undo the placeholder.

All the values required_cmsis_nn_buffer_sizes needs are already available locally in get*_replacement. If the size functions took inputs directly (e.g. an explicit ConvBufferSizeInputs dataclass), you could compute sizes before constructing the cortex_m op and emit allocs at their final size on one pass.

[Erik-Lundell]

I see your point, but I would like to keep the current design with the following arguments:

• node already perfectly captures the context needed to compute scratch buffer sizes. Introducing a new dataclass interface creates more boilerplate code with the risk of mismatching args.
• Importantly, node.meta["val"].shape is needed for conv, which requires either executing the node the calculate it (as is done now), or some duplication of logic to compute the shape from its args.
• cortex_m node creation is only done in one place in the pass (L513) , and the initialization happens directly after, so the logic is not too far separated.
• There is a check that the trailing args are allocs, so there is no silent mis-pairing. If alloc sizes were given in the wrong order, the runtime check would catch it.

I have pushed a commit to try to clarify the pattern though.

[digantdesai]

Nice!

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: Erik-Lundell / name: Erik Lundell (23c86c1, 264fc9e, 5f1cf38, 699e731, 7ad198a, 936ed4b, f801779)
• ✅ login: mansnils / name: Måns Nilsson (fae7030)

[Erik-Lundell]

This is done for conv, depthwise conv, transpose conv, and bmm.

Add scratch tensors to the operator signatures, which are then
assigned exir.memory.alloc. These allocs are automatically memory
planned by ExecuTorch.

Introduce required_cmsis_buffer_sizewhich computes the buffer
size from node properties + the Cortex-M configuration.
The function uses functions registered by target in
backends/cortex_m/passes/scratch_buffer_sizes.py
This is used to set the size of the allocs in ConvertToCortexMPass

Finally, modify the kernels to use the new scratch tensor instead
of allocating temporary memory. Add a new macro CORTEX_M_ENABLE_RUNTIME_CHECKS
to do a safety check that the aot computed buffer size is equal to the
buffer size computed at runtime. Use this when testing.

cc @psiddh@AdrianLundell@digantdesai@rascani@freddan80@per@zingo@oscarandersson8218@mansnils@Sebastian-Larsson@robell

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19636

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 3 Unrelated Failures

As of commit 7ad198a with merge base b73df0b ():

NEW FAILURES - The following jobs have failed:

• trunk / test-models-macos-cpu (llama2, xnnpack-quantization-delegation) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-qnn-optimum-model (fp32, pvt) / linux-job (gh)
  RuntimeError: Command docker exec -t 74145e3c8c2a31a529f88874bf2fc5b5e932ca4d6848ce98c8e9b592c10aa5c8 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• trunk / unittest-release / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Erik-Lundell]

This is a polished version of #16580

[rascani]

exir.passes.make_alloc_node is the canonical helper for emitting memory.alloc nodes. It's used by to_out_variant and to_scratch_op_pass use internally, and it sets meta["val"] and meta["tensor_meta"] on the alloc. The memory planner keys off meta["val"] / meta["tensor_meta"] to build the TensorSpec that drives lifetime analysis and arena placement.

[Erik-Lundell]

Thanks, I'll use that

[rascani]

IIUC, the current flow creates each scratch alloc with a placeholder shape, wires it into the cortex_m op's args, and then walks back through node.args[-(i+1)] in _set_scratch_buffer_size to mutate the alloc's shape once the size is known. This works, but the two-phase flow has a few drawbacks:

• The alloc -> size pairing is positional and implicit, split across two files. Adding a non-scratch trailing arg to any op signature silently mis-pairs.
• The UNINITIALIZED_ALLOC_ARGS sentinel + later mutation requires a reader to follow two hops to understand the alloc's actual size.
• _set_scratch_buffer_size exists only to undo the placeholder.

All the values required_cmsis_nn_buffer_sizes needs are already available locally in get*_replacement. If the size functions took inputs directly (e.g. an explicit ConvBufferSizeInputs dataclass), you could compute sizes before constructing the cortex_m op and emit allocs at their final size on one pass.

[Erik-Lundell]

I see your point, but I would like to keep the current design with the following arguments:

• node already perfectly captures the context needed to compute scratch buffer sizes. Introducing a new dataclass interface creates more boilerplate code with the risk of mismatching args.
• Importantly, node.meta["val"].shape is needed for conv, which requires either executing the node the calculate it (as is done now), or some duplication of logic to compute the shape from its args.
• cortex_m node creation is only done in one place in the pass (L513) , and the initialization happens directly after, so the logic is not too far separated.
• There is a check that the trailing args are allocs, so there is no silent mis-pairing. If alloc sizes were given in the wrong order, the runtime check would catch it.

I have pushed a commit to try to clarify the pattern though.

[digantdesai]

Nice!

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: Erik-Lundell / name: Erik Lundell (23c86c1, 264fc9e, 5f1cf38, 699e731, 7ad198a, 936ed4b, f801779)
• ✅ login: mansnils / name: Måns Nilsson (fae7030)

[Erik-Lundell]

This is done for conv, depthwise conv, transpose conv, and bmm.

Add scratch tensors to the operator signatures, which are then
assigned exir.memory.alloc. These allocs are automatically memory
planned by ExecuTorch.

Introduce required_cmsis_buffer_sizewhich computes the buffer
size from node properties + the Cortex-M configuration.
The function uses functions registered by target in
backends/cortex_m/passes/scratch_buffer_sizes.py
This is used to set the size of the allocs in ConvertToCortexMPass

Finally, modify the kernels to use the new scratch tensor instead
of allocating temporary memory. Add a new macro CORTEX_M_ENABLE_RUNTIME_CHECKS
to do a safety check that the aot computed buffer size is equal to the
buffer size computed at runtime. Use this when testing.

cc @psiddh@AdrianLundell@digantdesai@rascani@freddan80@per@zingo@oscarandersson8218@mansnils@Sebastian-Larsson@robell

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19636

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 3 Unrelated Failures

As of commit 7ad198a with merge base b73df0b ():

NEW FAILURES - The following jobs have failed:

• trunk / test-models-macos-cpu (llama2, xnnpack-quantization-delegation) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-qnn-optimum-model (fp32, pvt) / linux-job (gh)
  RuntimeError: Command docker exec -t 74145e3c8c2a31a529f88874bf2fc5b5e932ca4d6848ce98c8e9b592c10aa5c8 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• trunk / unittest-release / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Erik-Lundell]

This is a polished version of #16580

[rascani]

exir.passes.make_alloc_node is the canonical helper for emitting memory.alloc nodes. It's used by to_out_variant and to_scratch_op_pass use internally, and it sets meta["val"] and meta["tensor_meta"] on the alloc. The memory planner keys off meta["val"] / meta["tensor_meta"] to build the TensorSpec that drives lifetime analysis and arena placement.

[Erik-Lundell]

Thanks, I'll use that

[rascani]

IIUC, the current flow creates each scratch alloc with a placeholder shape, wires it into the cortex_m op's args, and then walks back through node.args[-(i+1)] in _set_scratch_buffer_size to mutate the alloc's shape once the size is known. This works, but the two-phase flow has a few drawbacks:

• The alloc -> size pairing is positional and implicit, split across two files. Adding a non-scratch trailing arg to any op signature silently mis-pairs.
• The UNINITIALIZED_ALLOC_ARGS sentinel + later mutation requires a reader to follow two hops to understand the alloc's actual size.
• _set_scratch_buffer_size exists only to undo the placeholder.

All the values required_cmsis_nn_buffer_sizes needs are already available locally in get*_replacement. If the size functions took inputs directly (e.g. an explicit ConvBufferSizeInputs dataclass), you could compute sizes before constructing the cortex_m op and emit allocs at their final size on one pass.

[Erik-Lundell]

I see your point, but I would like to keep the current design with the following arguments:

• node already perfectly captures the context needed to compute scratch buffer sizes. Introducing a new dataclass interface creates more boilerplate code with the risk of mismatching args.
• Importantly, node.meta["val"].shape is needed for conv, which requires either executing the node the calculate it (as is done now), or some duplication of logic to compute the shape from its args.
• cortex_m node creation is only done in one place in the pass (L513) , and the initialization happens directly after, so the logic is not too far separated.
• There is a check that the trailing args are allocs, so there is no silent mis-pairing. If alloc sizes were given in the wrong order, the runtime check would catch it.

I have pushed a commit to try to clarify the pattern though.

[digantdesai]

Nice!

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: Erik-Lundell / name: Erik Lundell (23c86c1, 264fc9e, 5f1cf38, 699e731, 7ad198a, 936ed4b, f801779)
• ✅ login: mansnils / name: Måns Nilsson (fae7030)

[Erik-Lundell]

This is done for conv, depthwise conv, transpose conv, and bmm.

Add scratch tensors to the operator signatures, which are then
assigned exir.memory.alloc. These allocs are automatically memory
planned by ExecuTorch.

Introduce required_cmsis_buffer_sizewhich computes the buffer
size from node properties + the Cortex-M configuration.
The function uses functions registered by target in
backends/cortex_m/passes/scratch_buffer_sizes.py
This is used to set the size of the allocs in ConvertToCortexMPass

Finally, modify the kernels to use the new scratch tensor instead
of allocating temporary memory. Add a new macro CORTEX_M_ENABLE_RUNTIME_CHECKS
to do a safety check that the aot computed buffer size is equal to the
buffer size computed at runtime. Use this when testing.

cc @psiddh@AdrianLundell@digantdesai@rascani@freddan80@per@zingo@oscarandersson8218@mansnils@Sebastian-Larsson@robell

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19636

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 3 Unrelated Failures

As of commit 7ad198a with merge base b73df0b ():

NEW FAILURES - The following jobs have failed:

• trunk / test-models-macos-cpu (llama2, xnnpack-quantization-delegation) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-qnn-optimum-model (fp32, pvt) / linux-job (gh)
  RuntimeError: Command docker exec -t 74145e3c8c2a31a529f88874bf2fc5b5e932ca4d6848ce98c8e9b592c10aa5c8 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• trunk / unittest-release / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Erik-Lundell]

This is a polished version of #16580

[rascani]

exir.passes.make_alloc_node is the canonical helper for emitting memory.alloc nodes. It's used by to_out_variant and to_scratch_op_pass use internally, and it sets meta["val"] and meta["tensor_meta"] on the alloc. The memory planner keys off meta["val"] / meta["tensor_meta"] to build the TensorSpec that drives lifetime analysis and arena placement.

[Erik-Lundell]

Thanks, I'll use that

[rascani]

IIUC, the current flow creates each scratch alloc with a placeholder shape, wires it into the cortex_m op's args, and then walks back through node.args[-(i+1)] in _set_scratch_buffer_size to mutate the alloc's shape once the size is known. This works, but the two-phase flow has a few drawbacks:

• The alloc -> size pairing is positional and implicit, split across two files. Adding a non-scratch trailing arg to any op signature silently mis-pairs.
• The UNINITIALIZED_ALLOC_ARGS sentinel + later mutation requires a reader to follow two hops to understand the alloc's actual size.
• _set_scratch_buffer_size exists only to undo the placeholder.

All the values required_cmsis_nn_buffer_sizes needs are already available locally in get*_replacement. If the size functions took inputs directly (e.g. an explicit ConvBufferSizeInputs dataclass), you could compute sizes before constructing the cortex_m op and emit allocs at their final size on one pass.

[Erik-Lundell]

I see your point, but I would like to keep the current design with the following arguments:

• node already perfectly captures the context needed to compute scratch buffer sizes. Introducing a new dataclass interface creates more boilerplate code with the risk of mismatching args.
• Importantly, node.meta["val"].shape is needed for conv, which requires either executing the node the calculate it (as is done now), or some duplication of logic to compute the shape from its args.
• cortex_m node creation is only done in one place in the pass (L513) , and the initialization happens directly after, so the logic is not too far separated.
• There is a check that the trailing args are allocs, so there is no silent mis-pairing. If alloc sizes were given in the wrong order, the runtime check would catch it.

I have pushed a commit to try to clarify the pattern though.

[digantdesai]

Nice!

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: Erik-Lundell / name: Erik Lundell (23c86c1, 264fc9e, 5f1cf38, 699e731, 7ad198a, 936ed4b, f801779)
• ✅ login: mansnils / name: Måns Nilsson (fae7030)

[Erik-Lundell]

This is done for conv, depthwise conv, transpose conv, and bmm.

Add scratch tensors to the operator signatures, which are then
assigned exir.memory.alloc. These allocs are automatically memory
planned by ExecuTorch.

Introduce required_cmsis_buffer_sizewhich computes the buffer
size from node properties + the Cortex-M configuration.
The function uses functions registered by target in
backends/cortex_m/passes/scratch_buffer_sizes.py
This is used to set the size of the allocs in ConvertToCortexMPass

Finally, modify the kernels to use the new scratch tensor instead
of allocating temporary memory. Add a new macro CORTEX_M_ENABLE_RUNTIME_CHECKS
to do a safety check that the aot computed buffer size is equal to the
buffer size computed at runtime. Use this when testing.

cc @psiddh@AdrianLundell@digantdesai@rascani@freddan80@per@zingo@oscarandersson8218@mansnils@Sebastian-Larsson@robell

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19636

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 3 Unrelated Failures

As of commit 7ad198a with merge base b73df0b ():

NEW FAILURES - The following jobs have failed:

• trunk / test-models-macos-cpu (llama2, xnnpack-quantization-delegation) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-qnn-optimum-model (fp32, pvt) / linux-job (gh)
  RuntimeError: Command docker exec -t 74145e3c8c2a31a529f88874bf2fc5b5e932ca4d6848ce98c8e9b592c10aa5c8 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• trunk / unittest-release / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Erik-Lundell]

This is a polished version of #16580

[rascani]

exir.passes.make_alloc_node is the canonical helper for emitting memory.alloc nodes. It's used by to_out_variant and to_scratch_op_pass use internally, and it sets meta["val"] and meta["tensor_meta"] on the alloc. The memory planner keys off meta["val"] / meta["tensor_meta"] to build the TensorSpec that drives lifetime analysis and arena placement.

[Erik-Lundell]

Thanks, I'll use that

[rascani]

IIUC, the current flow creates each scratch alloc with a placeholder shape, wires it into the cortex_m op's args, and then walks back through node.args[-(i+1)] in _set_scratch_buffer_size to mutate the alloc's shape once the size is known. This works, but the two-phase flow has a few drawbacks:

• The alloc -> size pairing is positional and implicit, split across two files. Adding a non-scratch trailing arg to any op signature silently mis-pairs.
• The UNINITIALIZED_ALLOC_ARGS sentinel + later mutation requires a reader to follow two hops to understand the alloc's actual size.
• _set_scratch_buffer_size exists only to undo the placeholder.

All the values required_cmsis_nn_buffer_sizes needs are already available locally in get*_replacement. If the size functions took inputs directly (e.g. an explicit ConvBufferSizeInputs dataclass), you could compute sizes before constructing the cortex_m op and emit allocs at their final size on one pass.

[Erik-Lundell]

I see your point, but I would like to keep the current design with the following arguments:

• node already perfectly captures the context needed to compute scratch buffer sizes. Introducing a new dataclass interface creates more boilerplate code with the risk of mismatching args.
• Importantly, node.meta["val"].shape is needed for conv, which requires either executing the node the calculate it (as is done now), or some duplication of logic to compute the shape from its args.
• cortex_m node creation is only done in one place in the pass (L513) , and the initialization happens directly after, so the logic is not too far separated.
• There is a check that the trailing args are allocs, so there is no silent mis-pairing. If alloc sizes were given in the wrong order, the runtime check would catch it.

I have pushed a commit to try to clarify the pattern though.

[digantdesai]

Nice!

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: Erik-Lundell / name: Erik Lundell (23c86c1, 264fc9e, 5f1cf38, 699e731, 7ad198a, 936ed4b, f801779)
• ✅ login: mansnils / name: Måns Nilsson (fae7030)

[Erik-Lundell]

This is done for conv, depthwise conv, transpose conv, and bmm.

Add scratch tensors to the operator signatures, which are then
assigned exir.memory.alloc. These allocs are automatically memory
planned by ExecuTorch.

Introduce required_cmsis_buffer_sizewhich computes the buffer
size from node properties + the Cortex-M configuration.
The function uses functions registered by target in
backends/cortex_m/passes/scratch_buffer_sizes.py
This is used to set the size of the allocs in ConvertToCortexMPass

Finally, modify the kernels to use the new scratch tensor instead
of allocating temporary memory. Add a new macro CORTEX_M_ENABLE_RUNTIME_CHECKS
to do a safety check that the aot computed buffer size is equal to the
buffer size computed at runtime. Use this when testing.

cc @psiddh@AdrianLundell@digantdesai@rascani@freddan80@per@zingo@oscarandersson8218@mansnils@Sebastian-Larsson@robell

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19636

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 3 Unrelated Failures

As of commit 7ad198a with merge base b73df0b ():

NEW FAILURES - The following jobs have failed:

• trunk / test-models-macos-cpu (llama2, xnnpack-quantization-delegation) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-qnn-optimum-model (fp32, pvt) / linux-job (gh)
  RuntimeError: Command docker exec -t 74145e3c8c2a31a529f88874bf2fc5b5e932ca4d6848ce98c8e9b592c10aa5c8 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• trunk / unittest-release / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Erik-Lundell]

This is a polished version of #16580

[rascani]

exir.passes.make_alloc_node is the canonical helper for emitting memory.alloc nodes. It's used by to_out_variant and to_scratch_op_pass use internally, and it sets meta["val"] and meta["tensor_meta"] on the alloc. The memory planner keys off meta["val"] / meta["tensor_meta"] to build the TensorSpec that drives lifetime analysis and arena placement.

[Erik-Lundell]

Thanks, I'll use that

[rascani]

IIUC, the current flow creates each scratch alloc with a placeholder shape, wires it into the cortex_m op's args, and then walks back through node.args[-(i+1)] in _set_scratch_buffer_size to mutate the alloc's shape once the size is known. This works, but the two-phase flow has a few drawbacks:

• The alloc -> size pairing is positional and implicit, split across two files. Adding a non-scratch trailing arg to any op signature silently mis-pairs.
• The UNINITIALIZED_ALLOC_ARGS sentinel + later mutation requires a reader to follow two hops to understand the alloc's actual size.
• _set_scratch_buffer_size exists only to undo the placeholder.

All the values required_cmsis_nn_buffer_sizes needs are already available locally in get*_replacement. If the size functions took inputs directly (e.g. an explicit ConvBufferSizeInputs dataclass), you could compute sizes before constructing the cortex_m op and emit allocs at their final size on one pass.

[Erik-Lundell]

I see your point, but I would like to keep the current design with the following arguments:

• node already perfectly captures the context needed to compute scratch buffer sizes. Introducing a new dataclass interface creates more boilerplate code with the risk of mismatching args.
• Importantly, node.meta["val"].shape is needed for conv, which requires either executing the node the calculate it (as is done now), or some duplication of logic to compute the shape from its args.
• cortex_m node creation is only done in one place in the pass (L513) , and the initialization happens directly after, so the logic is not too far separated.
• There is a check that the trailing args are allocs, so there is no silent mis-pairing. If alloc sizes were given in the wrong order, the runtime check would catch it.

I have pushed a commit to try to clarify the pattern though.

[digantdesai]

Nice!

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: Erik-Lundell / name: Erik Lundell (23c86c1, 264fc9e, 5f1cf38, 699e731, 7ad198a, 936ed4b, f801779)
• ✅ login: mansnils / name: Måns Nilsson (fae7030)

[Erik-Lundell]

This is done for conv, depthwise conv, transpose conv, and bmm.

Add scratch tensors to the operator signatures, which are then
assigned exir.memory.alloc. These allocs are automatically memory
planned by ExecuTorch.

Introduce required_cmsis_buffer_sizewhich computes the buffer
size from node properties + the Cortex-M configuration.
The function uses functions registered by target in
backends/cortex_m/passes/scratch_buffer_sizes.py
This is used to set the size of the allocs in ConvertToCortexMPass

Finally, modify the kernels to use the new scratch tensor instead
of allocating temporary memory. Add a new macro CORTEX_M_ENABLE_RUNTIME_CHECKS
to do a safety check that the aot computed buffer size is equal to the
buffer size computed at runtime. Use this when testing.

cc @psiddh@AdrianLundell@digantdesai@rascani@freddan80@per@zingo@oscarandersson8218@mansnils@Sebastian-Larsson@robell

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19636

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 3 Unrelated Failures

As of commit 7ad198a with merge base b73df0b ():

NEW FAILURES - The following jobs have failed:

• trunk / test-models-macos-cpu (llama2, xnnpack-quantization-delegation) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-qnn-optimum-model (fp32, pvt) / linux-job (gh)
  RuntimeError: Command docker exec -t 74145e3c8c2a31a529f88874bf2fc5b5e932ca4d6848ce98c8e9b592c10aa5c8 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.
• trunk / unittest-release / macos / macos-job (gh) (trunk failure)
  ##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Erik-Lundell]

This is a polished version of #16580

[rascani]

exir.passes.make_alloc_node is the canonical helper for emitting memory.alloc nodes. It's used by to_out_variant and to_scratch_op_pass use internally, and it sets meta["val"] and meta["tensor_meta"] on the alloc. The memory planner keys off meta["val"] / meta["tensor_meta"] to build the TensorSpec that drives lifetime analysis and arena placement.

[Erik-Lundell]

Thanks, I'll use that

[rascani]

IIUC, the current flow creates each scratch alloc with a placeholder shape, wires it into the cortex_m op's args, and then walks back through node.args[-(i+1)] in _set_scratch_buffer_size to mutate the alloc's shape once the size is known. This works, but the two-phase flow has a few drawbacks:

• The alloc -> size pairing is positional and implicit, split across two files. Adding a non-scratch trailing arg to any op signature silently mis-pairs.
• The UNINITIALIZED_ALLOC_ARGS sentinel + later mutation requires a reader to follow two hops to understand the alloc's actual size.
• _set_scratch_buffer_size exists only to undo the placeholder.

All the values required_cmsis_nn_buffer_sizes needs are already available locally in get*_replacement. If the size functions took inputs directly (e.g. an explicit ConvBufferSizeInputs dataclass), you could compute sizes before constructing the cortex_m op and emit allocs at their final size on one pass.

[Erik-Lundell]

I see your point, but I would like to keep the current design with the following arguments:

• node already perfectly captures the context needed to compute scratch buffer sizes. Introducing a new dataclass interface creates more boilerplate code with the risk of mismatching args.
• Importantly, node.meta["val"].shape is needed for conv, which requires either executing the node the calculate it (as is done now), or some duplication of logic to compute the shape from its args.
• cortex_m node creation is only done in one place in the pass (L513) , and the initialization happens directly after, so the logic is not too far separated.
• There is a check that the trailing args are allocs, so there is no silent mis-pairing. If alloc sizes were given in the wrong order, the runtime check would catch it.

I have pushed a commit to try to clarify the pattern though.

[digantdesai]

Nice!

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: Erik-Lundell / name: Erik Lundell (23c86c1, 264fc9e, 5f1cf38, 699e731, 7ad198a, 936ed4b, f801779)
• ✅ login: mansnils / name: Måns Nilsson (fae7030)