[Executorch] Introduce caching cpu memory allocator

[kimishpatel]

Stack from ghstack (oldest at bottom):

• Use caching allocator for runner (#15730) #15730
• [Executorch] Make module constructors uniform across #15729
• [Executorch] Use temp allocator for allocating scratch memory #15728
• ->[Executorch] Introduce caching cpu memory allocator #15611
• Reduce allocation overhead in quantized sdpa #15610

Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations

Differential Revision: D85532079

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15611

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 3 Unrelated Failures

As of commit 6b37360 with merge base a09a4b7 ():

NEW FAILURE - The following job has failed:

• trunk / test-mcu-cortex-m-backend / linux-job (gh)
  RuntimeError: Command docker exec -t 4f83ea84bb631196a18f6c4a4e7f31784cd515a40317378944dbcb29df077a4a /exec failed with exit code 1

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-large-v3-turbo, quantized-int4-weight-only) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-small, non-quantized) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / android / run-emulator (gh) (trunk failure)
  Timeout waiting for emulator to boot.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[barakugav]

@kimishpatel@metascroy hey there! im using cmake and im trying to use the new allocator in 1.1.0 but there are no options to pass to the root CMakeList.txt to include the memory allocator lib. Should it be added?

[kimishpatel]

Stack from ghstack (oldest at bottom):

• Use caching allocator for runner (#15730) #15730
• [Executorch] Make module constructors uniform across #15729
• [Executorch] Use temp allocator for allocating scratch memory #15728
• ->[Executorch] Introduce caching cpu memory allocator #15611
• Reduce allocation overhead in quantized sdpa #15610

Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations

Differential Revision: D85532079

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15611

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 3 Unrelated Failures

As of commit 6b37360 with merge base a09a4b7 ():

NEW FAILURE - The following job has failed:

• trunk / test-mcu-cortex-m-backend / linux-job (gh)
  RuntimeError: Command docker exec -t 4f83ea84bb631196a18f6c4a4e7f31784cd515a40317378944dbcb29df077a4a /exec failed with exit code 1

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-large-v3-turbo, quantized-int4-weight-only) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-small, non-quantized) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / android / run-emulator (gh) (trunk failure)
  Timeout waiting for emulator to boot.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[barakugav]

@kimishpatel@metascroy hey there! im using cmake and im trying to use the new allocator in 1.1.0 but there are no options to pass to the root CMakeList.txt to include the memory allocator lib. Should it be added?

[kimishpatel]

Stack from ghstack (oldest at bottom):

• Use caching allocator for runner (#15730) #15730
• [Executorch] Make module constructors uniform across #15729
• [Executorch] Use temp allocator for allocating scratch memory #15728
• ->[Executorch] Introduce caching cpu memory allocator #15611
• Reduce allocation overhead in quantized sdpa #15610

Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations

Differential Revision: D85532079

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15611

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 3 Unrelated Failures

As of commit 6b37360 with merge base a09a4b7 ():

NEW FAILURE - The following job has failed:

• trunk / test-mcu-cortex-m-backend / linux-job (gh)
  RuntimeError: Command docker exec -t 4f83ea84bb631196a18f6c4a4e7f31784cd515a40317378944dbcb29df077a4a /exec failed with exit code 1

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-large-v3-turbo, quantized-int4-weight-only) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-small, non-quantized) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / android / run-emulator (gh) (trunk failure)
  Timeout waiting for emulator to boot.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[barakugav]

@kimishpatel@metascroy hey there! im using cmake and im trying to use the new allocator in 1.1.0 but there are no options to pass to the root CMakeList.txt to include the memory allocator lib. Should it be added?

[kimishpatel]

Stack from ghstack (oldest at bottom):

• Use caching allocator for runner (#15730) #15730
• [Executorch] Make module constructors uniform across #15729
• [Executorch] Use temp allocator for allocating scratch memory #15728
• ->[Executorch] Introduce caching cpu memory allocator #15611
• Reduce allocation overhead in quantized sdpa #15610

Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations

Differential Revision: D85532079

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15611

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 3 Unrelated Failures

As of commit 6b37360 with merge base a09a4b7 ():

NEW FAILURE - The following job has failed:

• trunk / test-mcu-cortex-m-backend / linux-job (gh)
  RuntimeError: Command docker exec -t 4f83ea84bb631196a18f6c4a4e7f31784cd515a40317378944dbcb29df077a4a /exec failed with exit code 1

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-large-v3-turbo, quantized-int4-weight-only) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-small, non-quantized) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / android / run-emulator (gh) (trunk failure)
  Timeout waiting for emulator to boot.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[barakugav]

@kimishpatel@metascroy hey there! im using cmake and im trying to use the new allocator in 1.1.0 but there are no options to pass to the root CMakeList.txt to include the memory allocator lib. Should it be added?

[kimishpatel]

Stack from ghstack (oldest at bottom):

• Use caching allocator for runner (#15730) #15730
• [Executorch] Make module constructors uniform across #15729
• [Executorch] Use temp allocator for allocating scratch memory #15728
• ->[Executorch] Introduce caching cpu memory allocator #15611
• Reduce allocation overhead in quantized sdpa #15610

Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations

Differential Revision: D85532079

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15611

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 3 Unrelated Failures

As of commit 6b37360 with merge base a09a4b7 ():

NEW FAILURE - The following job has failed:

• trunk / test-mcu-cortex-m-backend / linux-job (gh)
  RuntimeError: Command docker exec -t 4f83ea84bb631196a18f6c4a4e7f31784cd515a40317378944dbcb29df077a4a /exec failed with exit code 1

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-large-v3-turbo, quantized-int4-weight-only) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-small, non-quantized) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / android / run-emulator (gh) (trunk failure)
  Timeout waiting for emulator to boot.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[barakugav]

@kimishpatel@metascroy hey there! im using cmake and im trying to use the new allocator in 1.1.0 but there are no options to pass to the root CMakeList.txt to include the memory allocator lib. Should it be added?

[kimishpatel]

Stack from ghstack (oldest at bottom):

• Use caching allocator for runner (#15730) #15730
• [Executorch] Make module constructors uniform across #15729
• [Executorch] Use temp allocator for allocating scratch memory #15728
• ->[Executorch] Introduce caching cpu memory allocator #15611
• Reduce allocation overhead in quantized sdpa #15610

Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations

Differential Revision: D85532079

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15611

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 3 Unrelated Failures

As of commit 6b37360 with merge base a09a4b7 ():

NEW FAILURE - The following job has failed:

• trunk / test-mcu-cortex-m-backend / linux-job (gh)
  RuntimeError: Command docker exec -t 4f83ea84bb631196a18f6c4a4e7f31784cd515a40317378944dbcb29df077a4a /exec failed with exit code 1

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-large-v3-turbo, quantized-int4-weight-only) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-small, non-quantized) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / android / run-emulator (gh) (trunk failure)
  Timeout waiting for emulator to boot.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[barakugav]

@kimishpatel@metascroy hey there! im using cmake and im trying to use the new allocator in 1.1.0 but there are no options to pass to the root CMakeList.txt to include the memory allocator lib. Should it be added?

[kimishpatel]

Stack from ghstack (oldest at bottom):

• Use caching allocator for runner (#15730) #15730
• [Executorch] Make module constructors uniform across #15729
• [Executorch] Use temp allocator for allocating scratch memory #15728
• ->[Executorch] Introduce caching cpu memory allocator #15611
• Reduce allocation overhead in quantized sdpa #15610

Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations

Differential Revision: D85532079

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15611

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 3 Unrelated Failures

As of commit 6b37360 with merge base a09a4b7 ():

NEW FAILURE - The following job has failed:

• trunk / test-mcu-cortex-m-backend / linux-job (gh)
  RuntimeError: Command docker exec -t 4f83ea84bb631196a18f6c4a4e7f31784cd515a40317378944dbcb29df077a4a /exec failed with exit code 1

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-large-v3-turbo, quantized-int4-weight-only) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-small, non-quantized) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / android / run-emulator (gh) (trunk failure)
  Timeout waiting for emulator to boot.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[barakugav]

@kimishpatel@metascroy hey there! im using cmake and im trying to use the new allocator in 1.1.0 but there are no options to pass to the root CMakeList.txt to include the memory allocator lib. Should it be added?

[kimishpatel]

Stack from ghstack (oldest at bottom):

• Use caching allocator for runner (#15730) #15730
• [Executorch] Make module constructors uniform across #15729
• [Executorch] Use temp allocator for allocating scratch memory #15728
• ->[Executorch] Introduce caching cpu memory allocator #15611
• Reduce allocation overhead in quantized sdpa #15610

Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations

Differential Revision: D85532079

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15611

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 3 Unrelated Failures

As of commit 6b37360 with merge base a09a4b7 ():

NEW FAILURE - The following job has failed:

• trunk / test-mcu-cortex-m-backend / linux-job (gh)
  RuntimeError: Command docker exec -t 4f83ea84bb631196a18f6c4a4e7f31784cd515a40317378944dbcb29df077a4a /exec failed with exit code 1

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-large-v3-turbo, quantized-int4-weight-only) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
• Test CUDA Builds / test-model-cuda-e2e (openai, whisper-small, non-quantized) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / android / run-emulator (gh) (trunk failure)
  Timeout waiting for emulator to boot.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[barakugav]

@kimishpatel@metascroy hey there! im using cmake and im trying to use the new allocator in 1.1.0 but there are no options to pass to the root CMakeList.txt to include the memory allocator lib. Should it be added?