Use caching allocator for runner (#15730) - #15730

Merged
meta-codesync[bot] merged 58 commits into
mainfrom
gh/kimishpatel/213/head
Apr 27, 2026
Merged

Use caching allocator for runner (#15730)#15730
meta-codesync[bot] merged 58 commits into
mainfrom
gh/kimishpatel/213/head

Conversation

@kimishpatel

@kimishpatelkimishpatel commented Nov 11, 2025•
edited by meta-codesync Bot
Loading

Copy link
Copy Markdown
Contributor

Summary:

We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.

No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport

Reviewed By: navsud, derekdixu

Differential Revision: D86120038

@kimishpatel
[Executorch] parallelize op_choose_qparams
3f789b8
When doing prefill for quantized kv cache, with large prefill length, parallelizing this op helps.
Differential Revision: [D84962234](https://our.internmc.facebook.com/intern/diff/D84962234/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962234/)!
[ghstack-poisoned]
@kimishpatel
[Executorch] Add simd path for op quantize
08dd980
Reason this doesnt directly use Vectorize class is because the equivalent APIs dont exist in Vectorize class
Differential Revision: [D84962236](https://our.internmc.facebook.com/intern/diff/D84962236/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962236/)!
[ghstack-poisoned]
@kimishpatel
[Executorch] Add multithreading for op_quantize
27fc8b1
As the title
Differential Revision: [D84962233](https://our.internmc.facebook.com/intern/diff/D84962233/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962233/)!
[ghstack-poisoned]
@kimishpatel
Reduce allocation overhead in quantized sdpa
ae61ab4
For small models dequantizing portions of v cache causes extra alloc overhead.
Probably a better way to handle this is to dequantize entire v cache outside the model
There isnt significant perf advantage from this yet but subsequent diffs will use caching allocator where this refactor help.
Differential Revision: [D85532077](https://our.internmc.facebook.com/intern/diff/D85532077/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Introduce caching cpu memory allocator
ea16e15
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
c3ed4b2
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
08ab552
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
dbf63cc
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
f9ce984
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
86c7c4b
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
0c23c32
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
68d76d3
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
79bb135
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Use temp allocator for allocating scratch memory
351a400
This allows us to leverage temp memory allocator and if that allocator is caching allocator it reduces the allocaiton overhead.
Differential Revision: [D85532076](https://our.internmc.facebook.com/intern/diff/D85532076/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Make module constructors uniform across
b4fdc22
Existing constructors dont compose well such that if you want data loader or data files constructor then you cannot get to override memory allocator.
Fix that.
Differential Revision: [D86120037](https://our.internmc.facebook.com/intern/diff/D86120037/)
[ghstack-poisoned]
This was referenced Nov 10, 2025
Merged
Merged
@pytorch-bot

pytorch-botBot commented Nov 11, 2025•
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15730

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

❌ 2 Cancelled Jobs, 2 Unrelated Failures

As of commit c75df37 with merge base eef7921 (image):

CANCELLED JOBS - The following jobs were cancelled. Please retry:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Nov 10, 2025
Merged
Merged
Merged
Merged
Merged
Merged
kimishpatel added a commit that referenced this pull request Nov 11, 2025
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
48ff224
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
ghstack-source-id: 322321964
Pull Request resolved: #15730
@meta-clameta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Nov 11, 2025
@meta-codesyncmeta-codesyncBot changed the title [Executorch][LLM] Use caching allocator for runnerUse caching allocator for runner (#15730)Mar 30, 2026
meta-codesyncBot pushed a commit that referenced this pull request Mar 30, 2026
@kimishpatel@facebook-github-bot
Use caching allocator for runner (#15730)
32ebe0f
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@meta-codesync
meta-codesyncBotforce-pushed the gh/kimishpatel/213/head branch from 3cd0176 to 32ebe0fCompareMarch 30, 2026 16:11
@meta-codesync

meta-codesyncBot commented Mar 30, 2026

Copy link
Copy Markdown
Contributor

@kimishpatel has exported this pull request. If you are a Meta employee, you can view the originating Diff in D86120038.

@kimishpatel@facebook-github-bot
Use caching allocator for runner (#15730)
6ebb435
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@meta-codesync
meta-codesyncBotforce-pushed the gh/kimishpatel/213/head branch from 32ebe0f to 6ebb435CompareApril 6, 2026 15:18
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
d759f09
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
11ec89c
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit that referenced this pull request Apr 15, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
165c945
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 367670359
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
e30bae0
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
704fb2e
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit that referenced this pull request Apr 16, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
818fb0a
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368353572
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
467774d
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
7359cf2
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit that referenced this pull request Apr 17, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
7f7921e
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368981082
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
d8b32c6
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
5b9bf5e
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit that referenced this pull request Apr 17, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
26e0b1f
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368983883
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
42830ac
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
3dd3158
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit that referenced this pull request Apr 20, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
c8ffd06
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 369818674
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
056a2a3
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
c75df37
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit that referenced this pull request Apr 24, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
4ef0252
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 372501554
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@meta-codesync
meta-codesyncBot merged commit cb4e5ae into mainApr 27, 2026
174 of 178 checks passed
@meta-codesync
meta-codesyncBot deleted the gh/kimishpatel/213/head branch April 27, 2026 21:40
@rascanirascani mentioned this pull request Apr 28, 2026
Merged
rascani added a commit that referenced this pull request Apr 29, 2026
@rascani
Add libextension_memory_allocator.a to executorch_llm Apple framework (…
5a206ab
…#19200)
### Summary
PR #15730 added a dependency from extension/llm/runner on
extension::CPUCachingAllocator (defined in
libextension_memory_allocator.a) but did not update
scripts/build_apple_frameworks.sh, so the executorch_llm iOS framework
was missing the symbol. This caused the Benchmark Tests target to fail
with `Undefined symbols: CPUCachingAllocator(unsigned int)`, breaking
the Apple/build-benchmark-app job on trunk.
Authored with the assistance of Claude (Anthropic).
### Test plan
CI
zeel2104 pushed a commit to zeel2104/executorch that referenced this pull request May 5, 2026
@kimishpatel@zeel2104
Use caching allocator for runner (pytorch#15730)
a54c8f7
Differential Revision: D86120038
Pull Request resolved: pytorch#15730
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@larryliu0820larryliu0820Awaiting requested review from larryliu0820larryliu0820 is a code owner
@kirklandsignkirklandsignAwaiting requested review from kirklandsignkirklandsign is a code owner
@jackzhxngjackzhxngAwaiting requested review from jackzhxng
@mergennachinmergennachinAwaiting requested review from mergennachinmergennachin is a code owner
+2 more reviewers
@derekxuderekxuderekxu approved these changes
@navsudnavsudnavsud approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.fb-exportedmeta-exportedStalePRs inactive for over 60 days

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kimishpatel@derekxu@navsud
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Use caching allocator for runner (#15730) - #15730
Merged
meta-codesync[bot] merged 58 commits into
mainfrom 
gh/kimishpatel/213/head
Apr 27, 2026
Merged
Use caching allocator for runner (#15730)#15730
meta-codesync[bot] merged 58 commits into
mainfrom 
gh/kimishpatel/213/head
Conversation
@kimishpatel
@kimishpatelkimishpatel
commented
Nov 11, 2025•

edited by meta-codesync
Bot
Loading

Copy link


Copy Markdown

Contributor
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.

ghstack-source-id: 328001114

exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@kimishpatel
[Executorch] parallelize op_choose_qparams
3f789b8
When doing prefill for quantized kv cache, with large prefill length, parallelizing this op helps.
Differential Revision: [D84962234](https://our.internmc.facebook.com/intern/diff/D84962234/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962234/)!
[ghstack-poisoned]
@kimishpatel
[Executorch] Add simd path for op quantize
08dd980
Reason this doesnt directly use Vectorize class is because the equivalent APIs dont exist in Vectorize class
Differential Revision: [D84962236](https://our.internmc.facebook.com/intern/diff/D84962236/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962236/)!
[ghstack-poisoned]
@kimishpatel
[Executorch] Add multithreading for op_quantize
27fc8b1
As the title
Differential Revision: [D84962233](https://our.internmc.facebook.com/intern/diff/D84962233/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962233/)!
[ghstack-poisoned]
@kimishpatel
Reduce allocation overhead in quantized sdpa
ae61ab4
For small models dequantizing portions of v cache causes extra alloc overhead.
Probably a better way to handle this is to dequantize entire v cache outside the model
There isnt significant perf advantage from this yet but subsequent diffs will use caching allocator where this refactor help.
Differential Revision: [D85532077](https://our.internmc.facebook.com/intern/diff/D85532077/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Introduce caching cpu memory allocator
ea16e15
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
c3ed4b2
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
08ab552
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
dbf63cc
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
f9ce984
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
86c7c4b
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
0c23c32
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
68d76d3
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
79bb135
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Use temp allocator for allocating scratch memory
351a400
This allows us to leverage temp memory allocator and if that allocator is caching allocator it reduces the allocaiton overhead.
Differential Revision: [D85532076](https://our.internmc.facebook.com/intern/diff/D85532076/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Make module constructors uniform across
b4fdc22
Existing constructors dont compose well such that if you want data loader or data files constructor then you cannot get to override memory allocator.
Fix that.
Differential Revision: [D86120037](https://our.internmc.facebook.com/intern/diff/D86120037/)
[ghstack-poisoned]

This was referenced Nov 10, 2025
 Merged

 Merged

@pytorch-bot
pytorch-botBot
commented
Nov 11, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15730
Note: Links to docs will display an error until the docs builds have been completed.
❗ 1 Active SEVs
There are 1 currently active SEVs. If your PR is affected, please view them below:
❌ 2 Cancelled Jobs, 2 Unrelated Failures
As of commit c75df37 with merge base eef7921 (image):
CANCELLED JOBS - The following jobs were cancelled. Please retry:
BROKEN TRUNK - The following jobs failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Nov 10, 2025
 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

kimishpatel added a commit
that referenced
this pull request
Nov 11, 2025
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
48ff224
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
ghstack-source-id: 322321964
Pull Request resolved: #15730
@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Nov 11, 2025
@meta-codesyncmeta-codesyncBot
changed the title
[Executorch][LLM] Use caching allocator for runnerUse caching allocator for runner (#15730)Mar 30, 2026
meta-codesyncBot
pushed a commit
that referenced
this pull request
Mar 30, 2026
@kimishpatel@facebook-github-bot
Use caching allocator for runner (#15730)
32ebe0f
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@meta-codesync
meta-codesyncBotforce-pushed
the
gh/kimishpatel/213/head
branch
from
3cd0176 to
32ebe0fCompareMarch 30, 2026 16:11
@meta-codesync
meta-codesyncBot
commented
Mar 30, 2026

Copy link


Copy Markdown

Contributor
@kimishpatel has exported this pull request. If you are a Meta employee, you can view the originating Diff in D86120038.
@kimishpatel@facebook-github-bot
Use caching allocator for runner (#15730)
6ebb435
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@meta-codesync
meta-codesyncBotforce-pushed
the
gh/kimishpatel/213/head
branch
from
32ebe0f to
6ebb435CompareApril 6, 2026 15:18
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
d759f09
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
11ec89c
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 15, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
165c945
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 367670359
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
e30bae0
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
704fb2e
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 16, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
818fb0a
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368353572
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
467774d
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
7359cf2
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 17, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
7f7921e
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368981082
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
d8b32c6
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
5b9bf5e
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 17, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
26e0b1f
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368983883
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
42830ac
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
3dd3158
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 20, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
c8ffd06
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 369818674
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
056a2a3
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
c75df37
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 24, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
4ef0252
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 372501554
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@meta-codesync
meta-codesyncBot
merged commit cb4e5ae
into
mainApr 27, 2026

174 of 178 checks passed

@meta-codesync
meta-codesyncBot
deleted the

gh/kimishpatel/213/head

branch
April 27, 2026 21:40
@rascanirascani
mentioned this pull request
Apr 28, 2026
 Merged

rascani added a commit
that referenced
this pull request
Apr 29, 2026
@rascani
Add libextension_memory_allocator.a to executorch_llm Apple framework (…
5a206ab
…#19200)
### Summary
PR #15730 added a dependency from extension/llm/runner on
extension::CPUCachingAllocator (defined in
libextension_memory_allocator.a) but did not update
scripts/build_apple_frameworks.sh, so the executorch_llm iOS framework
was missing the symbol. This caused the Benchmark Tests target to fail
with `Undefined symbols: CPUCachingAllocator(unsigned int)`, breaking
the Apple/build-benchmark-app job on trunk.
Authored with the assistance of Claude (Anthropic).
### Test plan
CI
zeel2104 pushed a commit
to zeel2104/executorch
that referenced
this pull request
May 5, 2026
@kimishpatel@zeel2104
Use caching allocator for runner (pytorch#15730)
a54c8f7
Differential Revision: D86120038
Pull Request resolved: pytorch#15730
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@larryliu0820larryliu0820Awaiting requested review from larryliu0820larryliu0820 is a code owner
@kirklandsignkirklandsignAwaiting requested review from kirklandsignkirklandsign is a code owner
@jackzhxngjackzhxngAwaiting requested review from jackzhxng
@mergennachinmergennachinAwaiting requested review from mergennachinmergennachin is a code owner

+2 more reviewers

@derekxuderekxuderekxu approved these changes
@navsudnavsudnavsud approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.fb-exportedmeta-exportedStalePRs inactive for over 60 days

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kimishpatel@derekxu@navsud

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Use caching allocator for runner (#15730) - #15730
Merged
meta-codesync[bot] merged 58 commits into
mainfrom 
gh/kimishpatel/213/head
Apr 27, 2026
Merged
Use caching allocator for runner (#15730)#15730
meta-codesync[bot] merged 58 commits into
mainfrom 
gh/kimishpatel/213/head
Conversation
@kimishpatel
@kimishpatelkimishpatel
commented
Nov 11, 2025•

edited by meta-codesync
Bot
Loading

Copy link


Copy Markdown

Contributor
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.

ghstack-source-id: 328001114

exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@kimishpatel
[Executorch] parallelize op_choose_qparams
3f789b8
When doing prefill for quantized kv cache, with large prefill length, parallelizing this op helps.
Differential Revision: [D84962234](https://our.internmc.facebook.com/intern/diff/D84962234/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962234/)!
[ghstack-poisoned]
@kimishpatel
[Executorch] Add simd path for op quantize
08dd980
Reason this doesnt directly use Vectorize class is because the equivalent APIs dont exist in Vectorize class
Differential Revision: [D84962236](https://our.internmc.facebook.com/intern/diff/D84962236/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962236/)!
[ghstack-poisoned]
@kimishpatel
[Executorch] Add multithreading for op_quantize
27fc8b1
As the title
Differential Revision: [D84962233](https://our.internmc.facebook.com/intern/diff/D84962233/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962233/)!
[ghstack-poisoned]
@kimishpatel
Reduce allocation overhead in quantized sdpa
ae61ab4
For small models dequantizing portions of v cache causes extra alloc overhead.
Probably a better way to handle this is to dequantize entire v cache outside the model
There isnt significant perf advantage from this yet but subsequent diffs will use caching allocator where this refactor help.
Differential Revision: [D85532077](https://our.internmc.facebook.com/intern/diff/D85532077/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Introduce caching cpu memory allocator
ea16e15
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
c3ed4b2
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
08ab552
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
dbf63cc
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
f9ce984
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
86c7c4b
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
0c23c32
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
68d76d3
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
79bb135
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Use temp allocator for allocating scratch memory
351a400
This allows us to leverage temp memory allocator and if that allocator is caching allocator it reduces the allocaiton overhead.
Differential Revision: [D85532076](https://our.internmc.facebook.com/intern/diff/D85532076/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Make module constructors uniform across
b4fdc22
Existing constructors dont compose well such that if you want data loader or data files constructor then you cannot get to override memory allocator.
Fix that.
Differential Revision: [D86120037](https://our.internmc.facebook.com/intern/diff/D86120037/)
[ghstack-poisoned]

This was referenced Nov 10, 2025
 Merged

 Merged

@pytorch-bot
pytorch-botBot
commented
Nov 11, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15730
Note: Links to docs will display an error until the docs builds have been completed.
❗ 1 Active SEVs
There are 1 currently active SEVs. If your PR is affected, please view them below:
❌ 2 Cancelled Jobs, 2 Unrelated Failures
As of commit c75df37 with merge base eef7921 (image):
CANCELLED JOBS - The following jobs were cancelled. Please retry:
BROKEN TRUNK - The following jobs failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Nov 10, 2025
 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

kimishpatel added a commit
that referenced
this pull request
Nov 11, 2025
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
48ff224
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
ghstack-source-id: 322321964
Pull Request resolved: #15730
@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Nov 11, 2025
@meta-codesyncmeta-codesyncBot
changed the title
[Executorch][LLM] Use caching allocator for runnerUse caching allocator for runner (#15730)Mar 30, 2026
meta-codesyncBot
pushed a commit
that referenced
this pull request
Mar 30, 2026
@kimishpatel@facebook-github-bot
Use caching allocator for runner (#15730)
32ebe0f
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@meta-codesync
meta-codesyncBotforce-pushed
the
gh/kimishpatel/213/head
branch
from
3cd0176 to
32ebe0fCompareMarch 30, 2026 16:11
@meta-codesync
meta-codesyncBot
commented
Mar 30, 2026

Copy link


Copy Markdown

Contributor
@kimishpatel has exported this pull request. If you are a Meta employee, you can view the originating Diff in D86120038.
@kimishpatel@facebook-github-bot
Use caching allocator for runner (#15730)
6ebb435
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@meta-codesync
meta-codesyncBotforce-pushed
the
gh/kimishpatel/213/head
branch
from
32ebe0f to
6ebb435CompareApril 6, 2026 15:18
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
d759f09
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
11ec89c
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 15, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
165c945
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 367670359
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
e30bae0
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
704fb2e
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 16, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
818fb0a
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368353572
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
467774d
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
7359cf2
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 17, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
7f7921e
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368981082
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
d8b32c6
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
5b9bf5e
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 17, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
26e0b1f
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368983883
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
42830ac
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
3dd3158
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 20, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
c8ffd06
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 369818674
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
056a2a3
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
c75df37
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 24, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
4ef0252
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 372501554
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@meta-codesync
meta-codesyncBot
merged commit cb4e5ae
into
mainApr 27, 2026

174 of 178 checks passed

@meta-codesync
meta-codesyncBot
deleted the

gh/kimishpatel/213/head

branch
April 27, 2026 21:40
@rascanirascani
mentioned this pull request
Apr 28, 2026
 Merged

rascani added a commit
that referenced
this pull request
Apr 29, 2026
@rascani
Add libextension_memory_allocator.a to executorch_llm Apple framework (…
5a206ab
…#19200)
### Summary
PR #15730 added a dependency from extension/llm/runner on
extension::CPUCachingAllocator (defined in
libextension_memory_allocator.a) but did not update
scripts/build_apple_frameworks.sh, so the executorch_llm iOS framework
was missing the symbol. This caused the Benchmark Tests target to fail
with `Undefined symbols: CPUCachingAllocator(unsigned int)`, breaking
the Apple/build-benchmark-app job on trunk.
Authored with the assistance of Claude (Anthropic).
### Test plan
CI
zeel2104 pushed a commit
to zeel2104/executorch
that referenced
this pull request
May 5, 2026
@kimishpatel@zeel2104
Use caching allocator for runner (pytorch#15730)
a54c8f7
Differential Revision: D86120038
Pull Request resolved: pytorch#15730
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@larryliu0820larryliu0820Awaiting requested review from larryliu0820larryliu0820 is a code owner
@kirklandsignkirklandsignAwaiting requested review from kirklandsignkirklandsign is a code owner
@jackzhxngjackzhxngAwaiting requested review from jackzhxng
@mergennachinmergennachinAwaiting requested review from mergennachinmergennachin is a code owner

+2 more reviewers

@derekxuderekxuderekxu approved these changes
@navsudnavsudnavsud approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.fb-exportedmeta-exportedStalePRs inactive for over 60 days

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kimishpatel@derekxu@navsud

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Use caching allocator for runner (#15730) - #15730
Merged
meta-codesync[bot] merged 58 commits into
mainfrom 
gh/kimishpatel/213/head
Apr 27, 2026
Merged
Use caching allocator for runner (#15730)#15730
meta-codesync[bot] merged 58 commits into
mainfrom 
gh/kimishpatel/213/head
Conversation
@kimishpatel
@kimishpatelkimishpatel
commented
Nov 11, 2025•

edited by meta-codesync
Bot
Loading

Copy link


Copy Markdown

Contributor
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.

ghstack-source-id: 328001114

exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@kimishpatel
[Executorch] parallelize op_choose_qparams
3f789b8
When doing prefill for quantized kv cache, with large prefill length, parallelizing this op helps.
Differential Revision: [D84962234](https://our.internmc.facebook.com/intern/diff/D84962234/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962234/)!
[ghstack-poisoned]
@kimishpatel
[Executorch] Add simd path for op quantize
08dd980
Reason this doesnt directly use Vectorize class is because the equivalent APIs dont exist in Vectorize class
Differential Revision: [D84962236](https://our.internmc.facebook.com/intern/diff/D84962236/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962236/)!
[ghstack-poisoned]
@kimishpatel
[Executorch] Add multithreading for op_quantize
27fc8b1
As the title
Differential Revision: [D84962233](https://our.internmc.facebook.com/intern/diff/D84962233/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962233/)!
[ghstack-poisoned]
@kimishpatel
Reduce allocation overhead in quantized sdpa
ae61ab4
For small models dequantizing portions of v cache causes extra alloc overhead.
Probably a better way to handle this is to dequantize entire v cache outside the model
There isnt significant perf advantage from this yet but subsequent diffs will use caching allocator where this refactor help.
Differential Revision: [D85532077](https://our.internmc.facebook.com/intern/diff/D85532077/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Introduce caching cpu memory allocator
ea16e15
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
c3ed4b2
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
08ab552
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
dbf63cc
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
f9ce984
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
86c7c4b
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
0c23c32
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
68d76d3
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
79bb135
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Use temp allocator for allocating scratch memory
351a400
This allows us to leverage temp memory allocator and if that allocator is caching allocator it reduces the allocaiton overhead.
Differential Revision: [D85532076](https://our.internmc.facebook.com/intern/diff/D85532076/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Make module constructors uniform across
b4fdc22
Existing constructors dont compose well such that if you want data loader or data files constructor then you cannot get to override memory allocator.
Fix that.
Differential Revision: [D86120037](https://our.internmc.facebook.com/intern/diff/D86120037/)
[ghstack-poisoned]

This was referenced Nov 10, 2025
 Merged

 Merged

@pytorch-bot
pytorch-botBot
commented
Nov 11, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15730
Note: Links to docs will display an error until the docs builds have been completed.
❗ 1 Active SEVs
There are 1 currently active SEVs. If your PR is affected, please view them below:
❌ 2 Cancelled Jobs, 2 Unrelated Failures
As of commit c75df37 with merge base eef7921 (image):
CANCELLED JOBS - The following jobs were cancelled. Please retry:
BROKEN TRUNK - The following jobs failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Nov 10, 2025
 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

kimishpatel added a commit
that referenced
this pull request
Nov 11, 2025
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
48ff224
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
ghstack-source-id: 322321964
Pull Request resolved: #15730
@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Nov 11, 2025
@meta-codesyncmeta-codesyncBot
changed the title
[Executorch][LLM] Use caching allocator for runnerUse caching allocator for runner (#15730)Mar 30, 2026
meta-codesyncBot
pushed a commit
that referenced
this pull request
Mar 30, 2026
@kimishpatel@facebook-github-bot
Use caching allocator for runner (#15730)
32ebe0f
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@meta-codesync
meta-codesyncBotforce-pushed
the
gh/kimishpatel/213/head
branch
from
3cd0176 to
32ebe0fCompareMarch 30, 2026 16:11
@meta-codesync
meta-codesyncBot
commented
Mar 30, 2026

Copy link


Copy Markdown

Contributor
@kimishpatel has exported this pull request. If you are a Meta employee, you can view the originating Diff in D86120038.
@kimishpatel@facebook-github-bot
Use caching allocator for runner (#15730)
6ebb435
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@meta-codesync
meta-codesyncBotforce-pushed
the
gh/kimishpatel/213/head
branch
from
32ebe0f to
6ebb435CompareApril 6, 2026 15:18
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
d759f09
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
11ec89c
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 15, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
165c945
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 367670359
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
e30bae0
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
704fb2e
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 16, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
818fb0a
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368353572
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
467774d
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
7359cf2
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 17, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
7f7921e
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368981082
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
d8b32c6
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
5b9bf5e
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 17, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
26e0b1f
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368983883
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
42830ac
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
3dd3158
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 20, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
c8ffd06
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 369818674
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
056a2a3
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
c75df37
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 24, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
4ef0252
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 372501554
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@meta-codesync
meta-codesyncBot
merged commit cb4e5ae
into
mainApr 27, 2026

174 of 178 checks passed

@meta-codesync
meta-codesyncBot
deleted the

gh/kimishpatel/213/head

branch
April 27, 2026 21:40
@rascanirascani
mentioned this pull request
Apr 28, 2026
 Merged

rascani added a commit
that referenced
this pull request
Apr 29, 2026
@rascani
Add libextension_memory_allocator.a to executorch_llm Apple framework (…
5a206ab
…#19200)
### Summary
PR #15730 added a dependency from extension/llm/runner on
extension::CPUCachingAllocator (defined in
libextension_memory_allocator.a) but did not update
scripts/build_apple_frameworks.sh, so the executorch_llm iOS framework
was missing the symbol. This caused the Benchmark Tests target to fail
with `Undefined symbols: CPUCachingAllocator(unsigned int)`, breaking
the Apple/build-benchmark-app job on trunk.
Authored with the assistance of Claude (Anthropic).
### Test plan
CI
zeel2104 pushed a commit
to zeel2104/executorch
that referenced
this pull request
May 5, 2026
@kimishpatel@zeel2104
Use caching allocator for runner (pytorch#15730)
a54c8f7
Differential Revision: D86120038
Pull Request resolved: pytorch#15730
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@larryliu0820larryliu0820Awaiting requested review from larryliu0820larryliu0820 is a code owner
@kirklandsignkirklandsignAwaiting requested review from kirklandsignkirklandsign is a code owner
@jackzhxngjackzhxngAwaiting requested review from jackzhxng
@mergennachinmergennachinAwaiting requested review from mergennachinmergennachin is a code owner

+2 more reviewers

@derekxuderekxuderekxu approved these changes
@navsudnavsudnavsud approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.fb-exportedmeta-exportedStalePRs inactive for over 60 days

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kimishpatel@derekxu@navsud

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Use caching allocator for runner (#15730) - #15730
Merged
meta-codesync[bot] merged 58 commits into
mainfrom 
gh/kimishpatel/213/head
Apr 27, 2026
Merged
Use caching allocator for runner (#15730)#15730
meta-codesync[bot] merged 58 commits into
mainfrom 
gh/kimishpatel/213/head
Conversation
@kimishpatel
@kimishpatelkimishpatel
commented
Nov 11, 2025•

edited by meta-codesync
Bot
Loading

Copy link


Copy Markdown

Contributor
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.

ghstack-source-id: 328001114

exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@kimishpatel
[Executorch] parallelize op_choose_qparams
3f789b8
When doing prefill for quantized kv cache, with large prefill length, parallelizing this op helps.
Differential Revision: [D84962234](https://our.internmc.facebook.com/intern/diff/D84962234/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962234/)!
[ghstack-poisoned]
@kimishpatel
[Executorch] Add simd path for op quantize
08dd980
Reason this doesnt directly use Vectorize class is because the equivalent APIs dont exist in Vectorize class
Differential Revision: [D84962236](https://our.internmc.facebook.com/intern/diff/D84962236/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962236/)!
[ghstack-poisoned]
@kimishpatel
[Executorch] Add multithreading for op_quantize
27fc8b1
As the title
Differential Revision: [D84962233](https://our.internmc.facebook.com/intern/diff/D84962233/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962233/)!
[ghstack-poisoned]
@kimishpatel
Reduce allocation overhead in quantized sdpa
ae61ab4
For small models dequantizing portions of v cache causes extra alloc overhead.
Probably a better way to handle this is to dequantize entire v cache outside the model
There isnt significant perf advantage from this yet but subsequent diffs will use caching allocator where this refactor help.
Differential Revision: [D85532077](https://our.internmc.facebook.com/intern/diff/D85532077/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Introduce caching cpu memory allocator
ea16e15
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
c3ed4b2
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
08ab552
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
dbf63cc
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
f9ce984
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
86c7c4b
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
0c23c32
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
68d76d3
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
79bb135
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Use temp allocator for allocating scratch memory
351a400
This allows us to leverage temp memory allocator and if that allocator is caching allocator it reduces the allocaiton overhead.
Differential Revision: [D85532076](https://our.internmc.facebook.com/intern/diff/D85532076/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Make module constructors uniform across
b4fdc22
Existing constructors dont compose well such that if you want data loader or data files constructor then you cannot get to override memory allocator.
Fix that.
Differential Revision: [D86120037](https://our.internmc.facebook.com/intern/diff/D86120037/)
[ghstack-poisoned]

This was referenced Nov 10, 2025
 Merged

 Merged

@pytorch-bot
pytorch-botBot
commented
Nov 11, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15730
Note: Links to docs will display an error until the docs builds have been completed.
❗ 1 Active SEVs
There are 1 currently active SEVs. If your PR is affected, please view them below:
❌ 2 Cancelled Jobs, 2 Unrelated Failures
As of commit c75df37 with merge base eef7921 (image):
CANCELLED JOBS - The following jobs were cancelled. Please retry:
BROKEN TRUNK - The following jobs failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Nov 10, 2025
 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

kimishpatel added a commit
that referenced
this pull request
Nov 11, 2025
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
48ff224
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
ghstack-source-id: 322321964
Pull Request resolved: #15730
@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Nov 11, 2025
@meta-codesyncmeta-codesyncBot
changed the title
[Executorch][LLM] Use caching allocator for runnerUse caching allocator for runner (#15730)Mar 30, 2026
meta-codesyncBot
pushed a commit
that referenced
this pull request
Mar 30, 2026
@kimishpatel@facebook-github-bot
Use caching allocator for runner (#15730)
32ebe0f
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@meta-codesync
meta-codesyncBotforce-pushed
the
gh/kimishpatel/213/head
branch
from
3cd0176 to
32ebe0fCompareMarch 30, 2026 16:11
@meta-codesync
meta-codesyncBot
commented
Mar 30, 2026

Copy link


Copy Markdown

Contributor
@kimishpatel has exported this pull request. If you are a Meta employee, you can view the originating Diff in D86120038.
@kimishpatel@facebook-github-bot
Use caching allocator for runner (#15730)
6ebb435
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@meta-codesync
meta-codesyncBotforce-pushed
the
gh/kimishpatel/213/head
branch
from
32ebe0f to
6ebb435CompareApril 6, 2026 15:18
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
d759f09
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
11ec89c
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 15, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
165c945
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 367670359
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
e30bae0
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
704fb2e
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 16, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
818fb0a
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368353572
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
467774d
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
7359cf2
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 17, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
7f7921e
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368981082
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
d8b32c6
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
5b9bf5e
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 17, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
26e0b1f
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368983883
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
42830ac
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
3dd3158
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 20, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
c8ffd06
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 369818674
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
056a2a3
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
c75df37
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 24, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
4ef0252
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 372501554
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@meta-codesync
meta-codesyncBot
merged commit cb4e5ae
into
mainApr 27, 2026

174 of 178 checks passed

@meta-codesync
meta-codesyncBot
deleted the

gh/kimishpatel/213/head

branch
April 27, 2026 21:40
@rascanirascani
mentioned this pull request
Apr 28, 2026
 Merged

rascani added a commit
that referenced
this pull request
Apr 29, 2026
@rascani
Add libextension_memory_allocator.a to executorch_llm Apple framework (…
5a206ab
…#19200)
### Summary
PR #15730 added a dependency from extension/llm/runner on
extension::CPUCachingAllocator (defined in
libextension_memory_allocator.a) but did not update
scripts/build_apple_frameworks.sh, so the executorch_llm iOS framework
was missing the symbol. This caused the Benchmark Tests target to fail
with `Undefined symbols: CPUCachingAllocator(unsigned int)`, breaking
the Apple/build-benchmark-app job on trunk.
Authored with the assistance of Claude (Anthropic).
### Test plan
CI
zeel2104 pushed a commit
to zeel2104/executorch
that referenced
this pull request
May 5, 2026
@kimishpatel@zeel2104
Use caching allocator for runner (pytorch#15730)
a54c8f7
Differential Revision: D86120038
Pull Request resolved: pytorch#15730
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@larryliu0820larryliu0820Awaiting requested review from larryliu0820larryliu0820 is a code owner
@kirklandsignkirklandsignAwaiting requested review from kirklandsignkirklandsign is a code owner
@jackzhxngjackzhxngAwaiting requested review from jackzhxng
@mergennachinmergennachinAwaiting requested review from mergennachinmergennachin is a code owner

+2 more reviewers

@derekxuderekxuderekxu approved these changes
@navsudnavsudnavsud approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.fb-exportedmeta-exportedStalePRs inactive for over 60 days

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kimishpatel@derekxu@navsud

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Use caching allocator for runner (#15730) - #15730
Merged
meta-codesync[bot] merged 58 commits into
mainfrom 
gh/kimishpatel/213/head
Apr 27, 2026
Merged
Use caching allocator for runner (#15730)#15730
meta-codesync[bot] merged 58 commits into
mainfrom 
gh/kimishpatel/213/head
Conversation
@kimishpatel
@kimishpatelkimishpatel
commented
Nov 11, 2025•

edited by meta-codesync
Bot
Loading

Copy link


Copy Markdown

Contributor
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.

ghstack-source-id: 328001114

exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@kimishpatel
[Executorch] parallelize op_choose_qparams
3f789b8
When doing prefill for quantized kv cache, with large prefill length, parallelizing this op helps.
Differential Revision: [D84962234](https://our.internmc.facebook.com/intern/diff/D84962234/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962234/)!
[ghstack-poisoned]
@kimishpatel
[Executorch] Add simd path for op quantize
08dd980
Reason this doesnt directly use Vectorize class is because the equivalent APIs dont exist in Vectorize class
Differential Revision: [D84962236](https://our.internmc.facebook.com/intern/diff/D84962236/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962236/)!
[ghstack-poisoned]
@kimishpatel
[Executorch] Add multithreading for op_quantize
27fc8b1
As the title
Differential Revision: [D84962233](https://our.internmc.facebook.com/intern/diff/D84962233/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962233/)!
[ghstack-poisoned]
@kimishpatel
Reduce allocation overhead in quantized sdpa
ae61ab4
For small models dequantizing portions of v cache causes extra alloc overhead.
Probably a better way to handle this is to dequantize entire v cache outside the model
There isnt significant perf advantage from this yet but subsequent diffs will use caching allocator where this refactor help.
Differential Revision: [D85532077](https://our.internmc.facebook.com/intern/diff/D85532077/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Introduce caching cpu memory allocator
ea16e15
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
c3ed4b2
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
08ab552
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
dbf63cc
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
f9ce984
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
86c7c4b
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
0c23c32
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
68d76d3
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
79bb135
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Use temp allocator for allocating scratch memory
351a400
This allows us to leverage temp memory allocator and if that allocator is caching allocator it reduces the allocaiton overhead.
Differential Revision: [D85532076](https://our.internmc.facebook.com/intern/diff/D85532076/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Make module constructors uniform across
b4fdc22
Existing constructors dont compose well such that if you want data loader or data files constructor then you cannot get to override memory allocator.
Fix that.
Differential Revision: [D86120037](https://our.internmc.facebook.com/intern/diff/D86120037/)
[ghstack-poisoned]

This was referenced Nov 10, 2025
 Merged

 Merged

@pytorch-bot
pytorch-botBot
commented
Nov 11, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15730
Note: Links to docs will display an error until the docs builds have been completed.
❗ 1 Active SEVs
There are 1 currently active SEVs. If your PR is affected, please view them below:
❌ 2 Cancelled Jobs, 2 Unrelated Failures
As of commit c75df37 with merge base eef7921 (image):
CANCELLED JOBS - The following jobs were cancelled. Please retry:
BROKEN TRUNK - The following jobs failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Nov 10, 2025
 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

kimishpatel added a commit
that referenced
this pull request
Nov 11, 2025
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
48ff224
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
ghstack-source-id: 322321964
Pull Request resolved: #15730
@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Nov 11, 2025
@meta-codesyncmeta-codesyncBot
changed the title
[Executorch][LLM] Use caching allocator for runnerUse caching allocator for runner (#15730)Mar 30, 2026
meta-codesyncBot
pushed a commit
that referenced
this pull request
Mar 30, 2026
@kimishpatel@facebook-github-bot
Use caching allocator for runner (#15730)
32ebe0f
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@meta-codesync
meta-codesyncBotforce-pushed
the
gh/kimishpatel/213/head
branch
from
3cd0176 to
32ebe0fCompareMarch 30, 2026 16:11
@meta-codesync
meta-codesyncBot
commented
Mar 30, 2026

Copy link


Copy Markdown

Contributor
@kimishpatel has exported this pull request. If you are a Meta employee, you can view the originating Diff in D86120038.
@kimishpatel@facebook-github-bot
Use caching allocator for runner (#15730)
6ebb435
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@meta-codesync
meta-codesyncBotforce-pushed
the
gh/kimishpatel/213/head
branch
from
32ebe0f to
6ebb435CompareApril 6, 2026 15:18
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
d759f09
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
11ec89c
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 15, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
165c945
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 367670359
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
e30bae0
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
704fb2e
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 16, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
818fb0a
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368353572
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
467774d
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
7359cf2
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 17, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
7f7921e
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368981082
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
d8b32c6
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
5b9bf5e
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 17, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
26e0b1f
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368983883
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
42830ac
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
3dd3158
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 20, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
c8ffd06
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 369818674
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
056a2a3
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
c75df37
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 24, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
4ef0252
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 372501554
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@meta-codesync
meta-codesyncBot
merged commit cb4e5ae
into
mainApr 27, 2026

174 of 178 checks passed

@meta-codesync
meta-codesyncBot
deleted the

gh/kimishpatel/213/head

branch
April 27, 2026 21:40
@rascanirascani
mentioned this pull request
Apr 28, 2026
 Merged

rascani added a commit
that referenced
this pull request
Apr 29, 2026
@rascani
Add libextension_memory_allocator.a to executorch_llm Apple framework (…
5a206ab
…#19200)
### Summary
PR #15730 added a dependency from extension/llm/runner on
extension::CPUCachingAllocator (defined in
libextension_memory_allocator.a) but did not update
scripts/build_apple_frameworks.sh, so the executorch_llm iOS framework
was missing the symbol. This caused the Benchmark Tests target to fail
with `Undefined symbols: CPUCachingAllocator(unsigned int)`, breaking
the Apple/build-benchmark-app job on trunk.
Authored with the assistance of Claude (Anthropic).
### Test plan
CI
zeel2104 pushed a commit
to zeel2104/executorch
that referenced
this pull request
May 5, 2026
@kimishpatel@zeel2104
Use caching allocator for runner (pytorch#15730)
a54c8f7
Differential Revision: D86120038
Pull Request resolved: pytorch#15730
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@larryliu0820larryliu0820Awaiting requested review from larryliu0820larryliu0820 is a code owner
@kirklandsignkirklandsignAwaiting requested review from kirklandsignkirklandsign is a code owner
@jackzhxngjackzhxngAwaiting requested review from jackzhxng
@mergennachinmergennachinAwaiting requested review from mergennachinmergennachin is a code owner

+2 more reviewers

@derekxuderekxuderekxu approved these changes
@navsudnavsudnavsud approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.fb-exportedmeta-exportedStalePRs inactive for over 60 days

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kimishpatel@derekxu@navsud

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Use caching allocator for runner (#15730) - #15730
Merged
meta-codesync[bot] merged 58 commits into
mainfrom 
gh/kimishpatel/213/head
Apr 27, 2026
Merged
Use caching allocator for runner (#15730)#15730
meta-codesync[bot] merged 58 commits into
mainfrom 
gh/kimishpatel/213/head
Conversation
@kimishpatel
@kimishpatelkimishpatel
commented
Nov 11, 2025•

edited by meta-codesync
Bot
Loading

Copy link


Copy Markdown

Contributor
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.

ghstack-source-id: 328001114

exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@kimishpatel
[Executorch] parallelize op_choose_qparams
3f789b8
When doing prefill for quantized kv cache, with large prefill length, parallelizing this op helps.
Differential Revision: [D84962234](https://our.internmc.facebook.com/intern/diff/D84962234/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962234/)!
[ghstack-poisoned]
@kimishpatel
[Executorch] Add simd path for op quantize
08dd980
Reason this doesnt directly use Vectorize class is because the equivalent APIs dont exist in Vectorize class
Differential Revision: [D84962236](https://our.internmc.facebook.com/intern/diff/D84962236/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962236/)!
[ghstack-poisoned]
@kimishpatel
[Executorch] Add multithreading for op_quantize
27fc8b1
As the title
Differential Revision: [D84962233](https://our.internmc.facebook.com/intern/diff/D84962233/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962233/)!
[ghstack-poisoned]
@kimishpatel
Reduce allocation overhead in quantized sdpa
ae61ab4
For small models dequantizing portions of v cache causes extra alloc overhead.
Probably a better way to handle this is to dequantize entire v cache outside the model
There isnt significant perf advantage from this yet but subsequent diffs will use caching allocator where this refactor help.
Differential Revision: [D85532077](https://our.internmc.facebook.com/intern/diff/D85532077/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Introduce caching cpu memory allocator
ea16e15
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
c3ed4b2
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
08ab552
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
dbf63cc
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
f9ce984
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
86c7c4b
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
0c23c32
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
68d76d3
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
79bb135
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Use temp allocator for allocating scratch memory
351a400
This allows us to leverage temp memory allocator and if that allocator is caching allocator it reduces the allocaiton overhead.
Differential Revision: [D85532076](https://our.internmc.facebook.com/intern/diff/D85532076/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Make module constructors uniform across
b4fdc22
Existing constructors dont compose well such that if you want data loader or data files constructor then you cannot get to override memory allocator.
Fix that.
Differential Revision: [D86120037](https://our.internmc.facebook.com/intern/diff/D86120037/)
[ghstack-poisoned]

This was referenced Nov 10, 2025
 Merged

 Merged

@pytorch-bot
pytorch-botBot
commented
Nov 11, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15730
Note: Links to docs will display an error until the docs builds have been completed.
❗ 1 Active SEVs
There are 1 currently active SEVs. If your PR is affected, please view them below:
❌ 2 Cancelled Jobs, 2 Unrelated Failures
As of commit c75df37 with merge base eef7921 (image):
CANCELLED JOBS - The following jobs were cancelled. Please retry:
BROKEN TRUNK - The following jobs failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Nov 10, 2025
 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

kimishpatel added a commit
that referenced
this pull request
Nov 11, 2025
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
48ff224
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
ghstack-source-id: 322321964
Pull Request resolved: #15730
@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Nov 11, 2025
@meta-codesyncmeta-codesyncBot
changed the title
[Executorch][LLM] Use caching allocator for runnerUse caching allocator for runner (#15730)Mar 30, 2026
meta-codesyncBot
pushed a commit
that referenced
this pull request
Mar 30, 2026
@kimishpatel@facebook-github-bot
Use caching allocator for runner (#15730)
32ebe0f
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@meta-codesync
meta-codesyncBotforce-pushed
the
gh/kimishpatel/213/head
branch
from
3cd0176 to
32ebe0fCompareMarch 30, 2026 16:11
@meta-codesync
meta-codesyncBot
commented
Mar 30, 2026

Copy link


Copy Markdown

Contributor
@kimishpatel has exported this pull request. If you are a Meta employee, you can view the originating Diff in D86120038.
@kimishpatel@facebook-github-bot
Use caching allocator for runner (#15730)
6ebb435
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@meta-codesync
meta-codesyncBotforce-pushed
the
gh/kimishpatel/213/head
branch
from
32ebe0f to
6ebb435CompareApril 6, 2026 15:18
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
d759f09
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
11ec89c
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 15, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
165c945
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 367670359
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
e30bae0
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
704fb2e
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 16, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
818fb0a
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368353572
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
467774d
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
7359cf2
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 17, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
7f7921e
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368981082
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
d8b32c6
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
5b9bf5e
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 17, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
26e0b1f
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368983883
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
42830ac
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
3dd3158
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 20, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
c8ffd06
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 369818674
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
056a2a3
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
c75df37
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 24, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
4ef0252
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 372501554
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@meta-codesync
meta-codesyncBot
merged commit cb4e5ae
into
mainApr 27, 2026

174 of 178 checks passed

@meta-codesync
meta-codesyncBot
deleted the

gh/kimishpatel/213/head

branch
April 27, 2026 21:40
@rascanirascani
mentioned this pull request
Apr 28, 2026
 Merged

rascani added a commit
that referenced
this pull request
Apr 29, 2026
@rascani
Add libextension_memory_allocator.a to executorch_llm Apple framework (…
5a206ab
…#19200)
### Summary
PR #15730 added a dependency from extension/llm/runner on
extension::CPUCachingAllocator (defined in
libextension_memory_allocator.a) but did not update
scripts/build_apple_frameworks.sh, so the executorch_llm iOS framework
was missing the symbol. This caused the Benchmark Tests target to fail
with `Undefined symbols: CPUCachingAllocator(unsigned int)`, breaking
the Apple/build-benchmark-app job on trunk.
Authored with the assistance of Claude (Anthropic).
### Test plan
CI
zeel2104 pushed a commit
to zeel2104/executorch
that referenced
this pull request
May 5, 2026
@kimishpatel@zeel2104
Use caching allocator for runner (pytorch#15730)
a54c8f7
Differential Revision: D86120038
Pull Request resolved: pytorch#15730
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@larryliu0820larryliu0820Awaiting requested review from larryliu0820larryliu0820 is a code owner
@kirklandsignkirklandsignAwaiting requested review from kirklandsignkirklandsign is a code owner
@jackzhxngjackzhxngAwaiting requested review from jackzhxng
@mergennachinmergennachinAwaiting requested review from mergennachinmergennachin is a code owner

+2 more reviewers

@derekxuderekxuderekxu approved these changes
@navsudnavsudnavsud approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.fb-exportedmeta-exportedStalePRs inactive for over 60 days

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kimishpatel@derekxu@navsud

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Use caching allocator for runner (#15730) - #15730
Merged
meta-codesync[bot] merged 58 commits into
mainfrom 
gh/kimishpatel/213/head
Apr 27, 2026
Merged
Use caching allocator for runner (#15730)#15730
meta-codesync[bot] merged 58 commits into
mainfrom 
gh/kimishpatel/213/head
Conversation
@kimishpatel
@kimishpatelkimishpatel
commented
Nov 11, 2025•

edited by meta-codesync
Bot
Loading

Copy link


Copy Markdown

Contributor
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.

ghstack-source-id: 328001114

exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@kimishpatel
[Executorch] parallelize op_choose_qparams
3f789b8
When doing prefill for quantized kv cache, with large prefill length, parallelizing this op helps.
Differential Revision: [D84962234](https://our.internmc.facebook.com/intern/diff/D84962234/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962234/)!
[ghstack-poisoned]
@kimishpatel
[Executorch] Add simd path for op quantize
08dd980
Reason this doesnt directly use Vectorize class is because the equivalent APIs dont exist in Vectorize class
Differential Revision: [D84962236](https://our.internmc.facebook.com/intern/diff/D84962236/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962236/)!
[ghstack-poisoned]
@kimishpatel
[Executorch] Add multithreading for op_quantize
27fc8b1
As the title
Differential Revision: [D84962233](https://our.internmc.facebook.com/intern/diff/D84962233/)
**NOTE FOR REVIEWERS**: This PR has internal Meta-specific changes or comments, please review them on [Phabricator](https://our.internmc.facebook.com/intern/diff/D84962233/)!
[ghstack-poisoned]
@kimishpatel
Reduce allocation overhead in quantized sdpa
ae61ab4
For small models dequantizing portions of v cache causes extra alloc overhead.
Probably a better way to handle this is to dequantize entire v cache outside the model
There isnt significant perf advantage from this yet but subsequent diffs will use caching allocator where this refactor help.
Differential Revision: [D85532077](https://our.internmc.facebook.com/intern/diff/D85532077/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Introduce caching cpu memory allocator
ea16e15
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
c3ed4b2
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
08ab552
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
dbf63cc
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
f9ce984
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
86c7c4b
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
0c23c32
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update base for Update on "[Executorch] Introduce caching cpu memory …
68d76d3
…allocator"
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
Update on "[Executorch] Introduce caching cpu memory allocator"
79bb135
Meant to use this for temp allocator for kernels. Specifically for sdpa, it seems that on iOS there is a significant overhead coming from allocations
Differential Revision: [D85532079](https://our.internmc.facebook.com/intern/diff/D85532079/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Use temp allocator for allocating scratch memory
351a400
This allows us to leverage temp memory allocator and if that allocator is caching allocator it reduces the allocaiton overhead.
Differential Revision: [D85532076](https://our.internmc.facebook.com/intern/diff/D85532076/)
[ghstack-poisoned]
@kimishpatel
[Executorch] Make module constructors uniform across
b4fdc22
Existing constructors dont compose well such that if you want data loader or data files constructor then you cannot get to override memory allocator.
Fix that.
Differential Revision: [D86120037](https://our.internmc.facebook.com/intern/diff/D86120037/)
[ghstack-poisoned]

This was referenced Nov 10, 2025
 Merged

 Merged

@pytorch-bot
pytorch-botBot
commented
Nov 11, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/15730
Note: Links to docs will display an error until the docs builds have been completed.
❗ 1 Active SEVs
There are 1 currently active SEVs. If your PR is affected, please view them below:
❌ 2 Cancelled Jobs, 2 Unrelated Failures
As of commit c75df37 with merge base eef7921 (image):
CANCELLED JOBS - The following jobs were cancelled. Please retry:
BROKEN TRUNK - The following jobs failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Nov 10, 2025
 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

kimishpatel added a commit
that referenced
this pull request
Nov 11, 2025
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
48ff224
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
ghstack-source-id: 322321964
Pull Request resolved: #15730
@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Nov 11, 2025
@meta-codesyncmeta-codesyncBot
changed the title
[Executorch][LLM] Use caching allocator for runnerUse caching allocator for runner (#15730)Mar 30, 2026
meta-codesyncBot
pushed a commit
that referenced
this pull request
Mar 30, 2026
@kimishpatel@facebook-github-bot
Use caching allocator for runner (#15730)
32ebe0f
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@meta-codesync
meta-codesyncBotforce-pushed
the
gh/kimishpatel/213/head
branch
from
3cd0176 to
32ebe0fCompareMarch 30, 2026 16:11
@meta-codesync
meta-codesyncBot
commented
Mar 30, 2026

Copy link


Copy Markdown

Contributor
@kimishpatel has exported this pull request. If you are a Meta employee, you can view the originating Diff in D86120038.
@kimishpatel@facebook-github-bot
Use caching allocator for runner (#15730)
6ebb435
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
@meta-codesync
meta-codesyncBotforce-pushed
the
gh/kimishpatel/213/head
branch
from
32ebe0f to
6ebb435CompareApril 6, 2026 15:18
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
d759f09
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
11ec89c
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 15, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
165c945
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 367670359
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
e30bae0
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
704fb2e
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 16, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
818fb0a
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368353572
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
467774d
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
7359cf2
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 17, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
7f7921e
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368981082
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
d8b32c6
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
5b9bf5e
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 17, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
26e0b1f
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 368983883
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
42830ac
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
3dd3158
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 20, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
c8ffd06
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 369818674
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@kimishpatel
Update base for Update on "Use caching allocator for runner (#15730)"
056a2a3
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
@kimishpatel
Update on "Use caching allocator for runner (#15730)"
c75df37
Summary:
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 328001114
exported-using-ghexport
Reviewed By: navsud, derekdixu
Differential Revision: D86120038
[ghstack-poisoned]
kimishpatel added a commit
that referenced
this pull request
Apr 24, 2026
@kimishpatel
[Executorch][LLM] Use caching allocator for runner
4ef0252
Pull Request resolved: #15730
We observed that on iOS it improves perf by 6% because SDPA op does temp allocations.
No significant difference on android though.
ghstack-source-id: 372501554
@exported-using-ghexport
Differential Revision: [D86120038](https://our.internmc.facebook.com/intern/diff/D86120038/)
@meta-codesync
meta-codesyncBot
merged commit cb4e5ae
into
mainApr 27, 2026

174 of 178 checks passed

@meta-codesync
meta-codesyncBot
deleted the

gh/kimishpatel/213/head

branch
April 27, 2026 21:40
@rascanirascani
mentioned this pull request
Apr 28, 2026
 Merged

rascani added a commit
that referenced
this pull request
Apr 29, 2026
@rascani
Add libextension_memory_allocator.a to executorch_llm Apple framework (…
5a206ab
…#19200)
### Summary
PR #15730 added a dependency from extension/llm/runner on
extension::CPUCachingAllocator (defined in
libextension_memory_allocator.a) but did not update
scripts/build_apple_frameworks.sh, so the executorch_llm iOS framework
was missing the symbol. This caused the Benchmark Tests target to fail
with `Undefined symbols: CPUCachingAllocator(unsigned int)`, breaking
the Apple/build-benchmark-app job on trunk.
Authored with the assistance of Claude (Anthropic).
### Test plan
CI
zeel2104 pushed a commit
to zeel2104/executorch
that referenced
this pull request
May 5, 2026
@kimishpatel@zeel2104
Use caching allocator for runner (pytorch#15730)
a54c8f7
Differential Revision: D86120038
Pull Request resolved: pytorch#15730
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

Copilot code reviewCopilot
Copilot review effort, defaults to Lite
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@larryliu0820larryliu0820Awaiting requested review from larryliu0820larryliu0820 is a code owner
@kirklandsignkirklandsignAwaiting requested review from kirklandsignkirklandsign is a code owner
@jackzhxngjackzhxngAwaiting requested review from jackzhxng
@mergennachinmergennachinAwaiting requested review from mergennachinmergennachin is a code owner

+2 more reviewers

@derekxuderekxuderekxu approved these changes
@navsudnavsudnavsud approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.fb-exportedmeta-exportedStalePRs inactive for over 60 days

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@kimishpatel@derekxu@navsud