[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill GEMM - #20800

Merged
meta-codesync[bot] merged 6 commits into
gh/JCNTH/15/basefrom
gh/JCNTH/15/head
Jul 9, 2026
Merged

[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill GEMM#20800
meta-codesync[bot] merged 6 commits into
gh/JCNTH/15/basefrom
gh/JCNTH/15/head

Conversation

@JCNTH

@JCNTHJCNTH commented Jul 9, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Stack from ghstack (oldest at bottom):

+46-56% end-to-end prefill tok/s over the shipped f16 steel q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).

Problem: the f16 steel prefill GEMM (and its packed-word-dequant variant pwdq) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.

Solution: an f16-accumulate variant of the pwdq kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with fma() (mirrors MLC's array<f16,16> reduction) and cast to f32 in the epilogue for the f32 output/bias.

Before (pwdq): f16 multiply, f32 accumulate.
After (pwdqf16acc): f16 multiply, f16 accumulate, f32 epilogue.

Implementation:

  • New ACC=half fork in the shared q4gsw_linear_gemm_steel.wgsl template (a shader_variants entry in q4gsw_linear_gemm_steel.yaml generates q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h) — no standalone shader file.
  • Opt-in via the enable_f16_accumulate_gemm runtime spec (a load-time BackendOption read in WebGPUBackend::init, threaded through WebGPUGraph::build(..., f16_accumulate_gemm) -> graph.f16_accumulate_gemm()), default OFF — no CMake option or compile flag.
  • When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and group_size % BK == 0; else the f32-accumulate pwdq / half / f32 kernels run (fail-closed).

Constraints: LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.

Co-authored-with: Claude Code.
@exported-using-ghexport

Differential Revision: D111163606

Differential Revision: D111163606

@JCNTH
Update
d29f101
[ghstack-poisoned]
@pytorch-bot

pytorch-botBot commented Jul 9, 2026•
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20800

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 12 Pending, 1 Unrelated Failure

As of commit 0b002a5 with merge base f4b01a8 (image):

NEW FAILURES - The following jobs have failed:

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

  • pull / android / build-android (gh) (trunk failure)
    ImportError: /opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/executorch/extension/training/pybindings/_training_lib.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE

This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Jul 9, 2026
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
Merged
@meta-clameta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 9, 2026
@JCNTHJCNTH mentioned this pull request Jul 9, 2026
Merged
@github-actions

github-actionsBot commented Jul 9, 2026

Copy link
Copy Markdown

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

@JCNTH
Update
1f9198a
[ghstack-poisoned]
@JCNTH
Update
8030226
[ghstack-poisoned]
SS-JIA
SS-JIA requested changes Jul 9, 2026
View reviewed changes

@SS-JIASS-JIA left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.

SS-JIA
SS-JIA approved these changes Jul 9, 2026
View reviewed changes

@SS-JIASS-JIA left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.

@JCNTH
Update
7465dc1
[ghstack-poisoned]
@JCNTH
Update
4af5d96
[ghstack-poisoned]
@JCNTH
Update
0b002a5
[ghstack-poisoned]
@meta-codesync
meta-codesyncBot merged commit 22e1ef3 into gh/JCNTH/15/baseJul 9, 2026
177 of 183 checks passed
@meta-codesync
meta-codesyncBot deleted the gh/JCNTH/15/head branch July 9, 2026 21:41
@pytorchbotpytorchbot mentioned this pull request Jul 9, 2026
Merged
JCNTH added a commit that referenced this pull request Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
63122c1
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit that referenced this pull request Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
8c7ddb8
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit that referenced this pull request Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e39a350
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit that referenced this pull request Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
248a688
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit that referenced this pull request Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
c677485
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit that referenced this pull request Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
d9c9db0
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit that referenced this pull request Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
0cc2d41
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit that referenced this pull request Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e27ac6d
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit that referenced this pull request Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
45ea327
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit that referenced this pull request Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
cb131ce
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit that referenced this pull request Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
953f569
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit that referenced this pull request Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e2c4392
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@SS-JIASS-JIASS-JIA approved these changes
@larryliu0820larryliu0820Awaiting requested review from larryliu0820
@kirklandsignkirklandsignAwaiting requested review from kirklandsign

Assignees

No one assigned

Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.meta-exported

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@JCNTH@SS-JIA@nil-is-all
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill GEMM - #20800
Merged
meta-codesync[bot] merged 6 commits into
gh/JCNTH/15/basefrom 
gh/JCNTH/15/head
Jul 9, 2026
Merged
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill GEMM#20800
meta-codesync[bot] merged 6 commits into
gh/JCNTH/15/basefrom 
gh/JCNTH/15/head
Conversation
@JCNTH
@JCNTHJCNTH
commented
Jul 9, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Stack from ghstack (oldest at bottom):
+46-56% end-to-end prefill tok/s over the shipped f16 steel q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).
Problem: the f16 steel prefill GEMM (and its packed-word-dequant variant pwdq) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
Solution: an f16-accumulate variant of the pwdq kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with fma() (mirrors MLC's array<f16,16> reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (pwdq): f16 multiply, f32 accumulate.

After (pwdqf16acc): f16 multiply, f16 accumulate, f32 epilogue.
Implementation:
  • New ACC=half fork in the shared q4gsw_linear_gemm_steel.wgsl template (a shader_variants entry in q4gsw_linear_gemm_steel.yaml generates q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h) — no standalone shader file.
  • Opt-in via the enable_f16_accumulate_gemm runtime spec (a load-time BackendOption read in WebGPUBackend::init, threaded through WebGPUGraph::build(..., f16_accumulate_gemm) -> graph.f16_accumulate_gemm()), default OFF — no CMake option or compile flag.
  • When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and group_size % BK == 0; else the f32-accumulate pwdq / half / f32 kernels run (fail-closed).
Constraints: LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.

@exported-using-ghexport
Differential Revision: D111163606
Differential Revision: D111163606
@JCNTH
Update
d29f101
[ghstack-poisoned]
@pytorch-bot
pytorch-botBot
commented
Jul 9, 2026•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20800
Note: Links to docs will display an error until the docs builds have been completed.
❌ 3 New Failures, 12 Pending, 1 Unrelated Failure
As of commit 0b002a5 with merge base f4b01a8 (image):
NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
  • pull / android / build-android (gh) (trunk failure)
    ImportError: /opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/executorch/extension/training/pybindings/_training_lib.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE
This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Jul 9, 2026
 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Jul 9, 2026
@JCNTHJCNTH
mentioned this pull request
Jul 9, 2026
 Merged

@github-actions
github-actionsBot
commented
Jul 9, 2026

Copy link


Copy Markdown

This PR needs a release notes: label
If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.
To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"
For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.
@JCNTH
Update
1f9198a
[ghstack-poisoned]
@JCNTH
Update
8030226
[ghstack-poisoned]
SS-JIA
SS-JIA
requested changes
Jul 9, 2026
View reviewed changes
@SS-JIASS-JIA
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.
SS-JIA
SS-JIA
approved these changes
Jul 9, 2026
View reviewed changes
@SS-JIASS-JIA
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.
@JCNTH
Update
7465dc1
[ghstack-poisoned]
@JCNTH
Update
4af5d96
[ghstack-poisoned]
@JCNTH
Update
0b002a5
[ghstack-poisoned]
@meta-codesync
meta-codesyncBot
merged commit 22e1ef3
into
gh/JCNTH/15/baseJul 9, 2026

177 of 183 checks passed

@meta-codesync
meta-codesyncBot
deleted the

gh/JCNTH/15/head

branch
July 9, 2026 21:41
@pytorchbotpytorchbot
mentioned this pull request
Jul 9, 2026
 Merged

JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
63122c1
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
8c7ddb8
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e39a350
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
248a688
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
c677485
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
d9c9db0
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
0cc2d41
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e27ac6d
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
45ea327
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
cb131ce
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
953f569
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e2c4392
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@SS-JIASS-JIASS-JIA approved these changes
@larryliu0820larryliu0820Awaiting requested review from larryliu0820
@kirklandsignkirklandsignAwaiting requested review from kirklandsign

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.meta-exported

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@JCNTH@SS-JIA@nil-is-all

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill GEMM - #20800
Merged
meta-codesync[bot] merged 6 commits into
gh/JCNTH/15/basefrom 
gh/JCNTH/15/head
Jul 9, 2026
Merged
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill GEMM#20800
meta-codesync[bot] merged 6 commits into
gh/JCNTH/15/basefrom 
gh/JCNTH/15/head
Conversation
@JCNTH
@JCNTHJCNTH
commented
Jul 9, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Stack from ghstack (oldest at bottom):
+46-56% end-to-end prefill tok/s over the shipped f16 steel q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).
Problem: the f16 steel prefill GEMM (and its packed-word-dequant variant pwdq) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
Solution: an f16-accumulate variant of the pwdq kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with fma() (mirrors MLC's array<f16,16> reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (pwdq): f16 multiply, f32 accumulate.

After (pwdqf16acc): f16 multiply, f16 accumulate, f32 epilogue.
Implementation:
  • New ACC=half fork in the shared q4gsw_linear_gemm_steel.wgsl template (a shader_variants entry in q4gsw_linear_gemm_steel.yaml generates q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h) — no standalone shader file.
  • Opt-in via the enable_f16_accumulate_gemm runtime spec (a load-time BackendOption read in WebGPUBackend::init, threaded through WebGPUGraph::build(..., f16_accumulate_gemm) -> graph.f16_accumulate_gemm()), default OFF — no CMake option or compile flag.
  • When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and group_size % BK == 0; else the f32-accumulate pwdq / half / f32 kernels run (fail-closed).
Constraints: LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.

@exported-using-ghexport
Differential Revision: D111163606
Differential Revision: D111163606
@JCNTH
Update
d29f101
[ghstack-poisoned]
@pytorch-bot
pytorch-botBot
commented
Jul 9, 2026•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20800
Note: Links to docs will display an error until the docs builds have been completed.
❌ 3 New Failures, 12 Pending, 1 Unrelated Failure
As of commit 0b002a5 with merge base f4b01a8 (image):
NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
  • pull / android / build-android (gh) (trunk failure)
    ImportError: /opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/executorch/extension/training/pybindings/_training_lib.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE
This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Jul 9, 2026
 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Jul 9, 2026
@JCNTHJCNTH
mentioned this pull request
Jul 9, 2026
 Merged

@github-actions
github-actionsBot
commented
Jul 9, 2026

Copy link


Copy Markdown

This PR needs a release notes: label
If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.
To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"
For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.
@JCNTH
Update
1f9198a
[ghstack-poisoned]
@JCNTH
Update
8030226
[ghstack-poisoned]
SS-JIA
SS-JIA
requested changes
Jul 9, 2026
View reviewed changes
@SS-JIASS-JIA
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.
SS-JIA
SS-JIA
approved these changes
Jul 9, 2026
View reviewed changes
@SS-JIASS-JIA
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.
@JCNTH
Update
7465dc1
[ghstack-poisoned]
@JCNTH
Update
4af5d96
[ghstack-poisoned]
@JCNTH
Update
0b002a5
[ghstack-poisoned]
@meta-codesync
meta-codesyncBot
merged commit 22e1ef3
into
gh/JCNTH/15/baseJul 9, 2026

177 of 183 checks passed

@meta-codesync
meta-codesyncBot
deleted the

gh/JCNTH/15/head

branch
July 9, 2026 21:41
@pytorchbotpytorchbot
mentioned this pull request
Jul 9, 2026
 Merged

JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
63122c1
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
8c7ddb8
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e39a350
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
248a688
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
c677485
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
d9c9db0
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
0cc2d41
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e27ac6d
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
45ea327
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
cb131ce
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
953f569
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e2c4392
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@SS-JIASS-JIASS-JIA approved these changes
@larryliu0820larryliu0820Awaiting requested review from larryliu0820
@kirklandsignkirklandsignAwaiting requested review from kirklandsign

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.meta-exported

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@JCNTH@SS-JIA@nil-is-all

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill GEMM - #20800
Merged
meta-codesync[bot] merged 6 commits into
gh/JCNTH/15/basefrom 
gh/JCNTH/15/head
Jul 9, 2026
Merged
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill GEMM#20800
meta-codesync[bot] merged 6 commits into
gh/JCNTH/15/basefrom 
gh/JCNTH/15/head
Conversation
@JCNTH
@JCNTHJCNTH
commented
Jul 9, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Stack from ghstack (oldest at bottom):
+46-56% end-to-end prefill tok/s over the shipped f16 steel q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).
Problem: the f16 steel prefill GEMM (and its packed-word-dequant variant pwdq) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
Solution: an f16-accumulate variant of the pwdq kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with fma() (mirrors MLC's array<f16,16> reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (pwdq): f16 multiply, f32 accumulate.

After (pwdqf16acc): f16 multiply, f16 accumulate, f32 epilogue.
Implementation:
  • New ACC=half fork in the shared q4gsw_linear_gemm_steel.wgsl template (a shader_variants entry in q4gsw_linear_gemm_steel.yaml generates q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h) — no standalone shader file.
  • Opt-in via the enable_f16_accumulate_gemm runtime spec (a load-time BackendOption read in WebGPUBackend::init, threaded through WebGPUGraph::build(..., f16_accumulate_gemm) -> graph.f16_accumulate_gemm()), default OFF — no CMake option or compile flag.
  • When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and group_size % BK == 0; else the f32-accumulate pwdq / half / f32 kernels run (fail-closed).
Constraints: LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.

@exported-using-ghexport
Differential Revision: D111163606
Differential Revision: D111163606
@JCNTH
Update
d29f101
[ghstack-poisoned]
@pytorch-bot
pytorch-botBot
commented
Jul 9, 2026•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20800
Note: Links to docs will display an error until the docs builds have been completed.
❌ 3 New Failures, 12 Pending, 1 Unrelated Failure
As of commit 0b002a5 with merge base f4b01a8 (image):
NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
  • pull / android / build-android (gh) (trunk failure)
    ImportError: /opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/executorch/extension/training/pybindings/_training_lib.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE
This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Jul 9, 2026
 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Jul 9, 2026
@JCNTHJCNTH
mentioned this pull request
Jul 9, 2026
 Merged

@github-actions
github-actionsBot
commented
Jul 9, 2026

Copy link


Copy Markdown

This PR needs a release notes: label
If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.
To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"
For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.
@JCNTH
Update
1f9198a
[ghstack-poisoned]
@JCNTH
Update
8030226
[ghstack-poisoned]
SS-JIA
SS-JIA
requested changes
Jul 9, 2026
View reviewed changes
@SS-JIASS-JIA
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.
SS-JIA
SS-JIA
approved these changes
Jul 9, 2026
View reviewed changes
@SS-JIASS-JIA
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.
@JCNTH
Update
7465dc1
[ghstack-poisoned]
@JCNTH
Update
4af5d96
[ghstack-poisoned]
@JCNTH
Update
0b002a5
[ghstack-poisoned]
@meta-codesync
meta-codesyncBot
merged commit 22e1ef3
into
gh/JCNTH/15/baseJul 9, 2026

177 of 183 checks passed

@meta-codesync
meta-codesyncBot
deleted the

gh/JCNTH/15/head

branch
July 9, 2026 21:41
@pytorchbotpytorchbot
mentioned this pull request
Jul 9, 2026
 Merged

JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
63122c1
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
8c7ddb8
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e39a350
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
248a688
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
c677485
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
d9c9db0
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
0cc2d41
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e27ac6d
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
45ea327
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
cb131ce
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
953f569
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e2c4392
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@SS-JIASS-JIASS-JIA approved these changes
@larryliu0820larryliu0820Awaiting requested review from larryliu0820
@kirklandsignkirklandsignAwaiting requested review from kirklandsign

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.meta-exported

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@JCNTH@SS-JIA@nil-is-all

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill GEMM - #20800
Merged
meta-codesync[bot] merged 6 commits into
gh/JCNTH/15/basefrom 
gh/JCNTH/15/head
Jul 9, 2026
Merged
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill GEMM#20800
meta-codesync[bot] merged 6 commits into
gh/JCNTH/15/basefrom 
gh/JCNTH/15/head
Conversation
@JCNTH
@JCNTHJCNTH
commented
Jul 9, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Stack from ghstack (oldest at bottom):
+46-56% end-to-end prefill tok/s over the shipped f16 steel q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).
Problem: the f16 steel prefill GEMM (and its packed-word-dequant variant pwdq) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
Solution: an f16-accumulate variant of the pwdq kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with fma() (mirrors MLC's array<f16,16> reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (pwdq): f16 multiply, f32 accumulate.

After (pwdqf16acc): f16 multiply, f16 accumulate, f32 epilogue.
Implementation:
  • New ACC=half fork in the shared q4gsw_linear_gemm_steel.wgsl template (a shader_variants entry in q4gsw_linear_gemm_steel.yaml generates q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h) — no standalone shader file.
  • Opt-in via the enable_f16_accumulate_gemm runtime spec (a load-time BackendOption read in WebGPUBackend::init, threaded through WebGPUGraph::build(..., f16_accumulate_gemm) -> graph.f16_accumulate_gemm()), default OFF — no CMake option or compile flag.
  • When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and group_size % BK == 0; else the f32-accumulate pwdq / half / f32 kernels run (fail-closed).
Constraints: LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.

@exported-using-ghexport
Differential Revision: D111163606
Differential Revision: D111163606
@JCNTH
Update
d29f101
[ghstack-poisoned]
@pytorch-bot
pytorch-botBot
commented
Jul 9, 2026•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20800
Note: Links to docs will display an error until the docs builds have been completed.
❌ 3 New Failures, 12 Pending, 1 Unrelated Failure
As of commit 0b002a5 with merge base f4b01a8 (image):
NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
  • pull / android / build-android (gh) (trunk failure)
    ImportError: /opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/executorch/extension/training/pybindings/_training_lib.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE
This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Jul 9, 2026
 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Jul 9, 2026
@JCNTHJCNTH
mentioned this pull request
Jul 9, 2026
 Merged

@github-actions
github-actionsBot
commented
Jul 9, 2026

Copy link


Copy Markdown

This PR needs a release notes: label
If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.
To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"
For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.
@JCNTH
Update
1f9198a
[ghstack-poisoned]
@JCNTH
Update
8030226
[ghstack-poisoned]
SS-JIA
SS-JIA
requested changes
Jul 9, 2026
View reviewed changes
@SS-JIASS-JIA
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.
SS-JIA
SS-JIA
approved these changes
Jul 9, 2026
View reviewed changes
@SS-JIASS-JIA
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.
@JCNTH
Update
7465dc1
[ghstack-poisoned]
@JCNTH
Update
4af5d96
[ghstack-poisoned]
@JCNTH
Update
0b002a5
[ghstack-poisoned]
@meta-codesync
meta-codesyncBot
merged commit 22e1ef3
into
gh/JCNTH/15/baseJul 9, 2026

177 of 183 checks passed

@meta-codesync
meta-codesyncBot
deleted the

gh/JCNTH/15/head

branch
July 9, 2026 21:41
@pytorchbotpytorchbot
mentioned this pull request
Jul 9, 2026
 Merged

JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
63122c1
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
8c7ddb8
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e39a350
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
248a688
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
c677485
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
d9c9db0
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
0cc2d41
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e27ac6d
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
45ea327
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
cb131ce
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
953f569
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e2c4392
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@SS-JIASS-JIASS-JIA approved these changes
@larryliu0820larryliu0820Awaiting requested review from larryliu0820
@kirklandsignkirklandsignAwaiting requested review from kirklandsign

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.meta-exported

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@JCNTH@SS-JIA@nil-is-all

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill GEMM - #20800
Merged
meta-codesync[bot] merged 6 commits into
gh/JCNTH/15/basefrom 
gh/JCNTH/15/head
Jul 9, 2026
Merged
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill GEMM#20800
meta-codesync[bot] merged 6 commits into
gh/JCNTH/15/basefrom 
gh/JCNTH/15/head
Conversation
@JCNTH
@JCNTHJCNTH
commented
Jul 9, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Stack from ghstack (oldest at bottom):
+46-56% end-to-end prefill tok/s over the shipped f16 steel q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).
Problem: the f16 steel prefill GEMM (and its packed-word-dequant variant pwdq) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
Solution: an f16-accumulate variant of the pwdq kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with fma() (mirrors MLC's array<f16,16> reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (pwdq): f16 multiply, f32 accumulate.

After (pwdqf16acc): f16 multiply, f16 accumulate, f32 epilogue.
Implementation:
  • New ACC=half fork in the shared q4gsw_linear_gemm_steel.wgsl template (a shader_variants entry in q4gsw_linear_gemm_steel.yaml generates q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h) — no standalone shader file.
  • Opt-in via the enable_f16_accumulate_gemm runtime spec (a load-time BackendOption read in WebGPUBackend::init, threaded through WebGPUGraph::build(..., f16_accumulate_gemm) -> graph.f16_accumulate_gemm()), default OFF — no CMake option or compile flag.
  • When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and group_size % BK == 0; else the f32-accumulate pwdq / half / f32 kernels run (fail-closed).
Constraints: LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.

@exported-using-ghexport
Differential Revision: D111163606
Differential Revision: D111163606
@JCNTH
Update
d29f101
[ghstack-poisoned]
@pytorch-bot
pytorch-botBot
commented
Jul 9, 2026•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20800
Note: Links to docs will display an error until the docs builds have been completed.
❌ 3 New Failures, 12 Pending, 1 Unrelated Failure
As of commit 0b002a5 with merge base f4b01a8 (image):
NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
  • pull / android / build-android (gh) (trunk failure)
    ImportError: /opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/executorch/extension/training/pybindings/_training_lib.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE
This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Jul 9, 2026
 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Jul 9, 2026
@JCNTHJCNTH
mentioned this pull request
Jul 9, 2026
 Merged

@github-actions
github-actionsBot
commented
Jul 9, 2026

Copy link


Copy Markdown

This PR needs a release notes: label
If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.
To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"
For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.
@JCNTH
Update
1f9198a
[ghstack-poisoned]
@JCNTH
Update
8030226
[ghstack-poisoned]
SS-JIA
SS-JIA
requested changes
Jul 9, 2026
View reviewed changes
@SS-JIASS-JIA
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.
SS-JIA
SS-JIA
approved these changes
Jul 9, 2026
View reviewed changes
@SS-JIASS-JIA
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.
@JCNTH
Update
7465dc1
[ghstack-poisoned]
@JCNTH
Update
4af5d96
[ghstack-poisoned]
@JCNTH
Update
0b002a5
[ghstack-poisoned]
@meta-codesync
meta-codesyncBot
merged commit 22e1ef3
into
gh/JCNTH/15/baseJul 9, 2026

177 of 183 checks passed

@meta-codesync
meta-codesyncBot
deleted the

gh/JCNTH/15/head

branch
July 9, 2026 21:41
@pytorchbotpytorchbot
mentioned this pull request
Jul 9, 2026
 Merged

JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
63122c1
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
8c7ddb8
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e39a350
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
248a688
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
c677485
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
d9c9db0
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
0cc2d41
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e27ac6d
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
45ea327
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
cb131ce
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
953f569
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e2c4392
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@SS-JIASS-JIASS-JIA approved these changes
@larryliu0820larryliu0820Awaiting requested review from larryliu0820
@kirklandsignkirklandsignAwaiting requested review from kirklandsign

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.meta-exported

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@JCNTH@SS-JIA@nil-is-all

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill GEMM - #20800
Merged
meta-codesync[bot] merged 6 commits into
gh/JCNTH/15/basefrom 
gh/JCNTH/15/head
Jul 9, 2026
Merged
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill GEMM#20800
meta-codesync[bot] merged 6 commits into
gh/JCNTH/15/basefrom 
gh/JCNTH/15/head
Conversation
@JCNTH
@JCNTHJCNTH
commented
Jul 9, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Stack from ghstack (oldest at bottom):
+46-56% end-to-end prefill tok/s over the shipped f16 steel q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).
Problem: the f16 steel prefill GEMM (and its packed-word-dequant variant pwdq) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
Solution: an f16-accumulate variant of the pwdq kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with fma() (mirrors MLC's array<f16,16> reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (pwdq): f16 multiply, f32 accumulate.

After (pwdqf16acc): f16 multiply, f16 accumulate, f32 epilogue.
Implementation:
  • New ACC=half fork in the shared q4gsw_linear_gemm_steel.wgsl template (a shader_variants entry in q4gsw_linear_gemm_steel.yaml generates q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h) — no standalone shader file.
  • Opt-in via the enable_f16_accumulate_gemm runtime spec (a load-time BackendOption read in WebGPUBackend::init, threaded through WebGPUGraph::build(..., f16_accumulate_gemm) -> graph.f16_accumulate_gemm()), default OFF — no CMake option or compile flag.
  • When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and group_size % BK == 0; else the f32-accumulate pwdq / half / f32 kernels run (fail-closed).
Constraints: LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.

@exported-using-ghexport
Differential Revision: D111163606
Differential Revision: D111163606
@JCNTH
Update
d29f101
[ghstack-poisoned]
@pytorch-bot
pytorch-botBot
commented
Jul 9, 2026•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20800
Note: Links to docs will display an error until the docs builds have been completed.
❌ 3 New Failures, 12 Pending, 1 Unrelated Failure
As of commit 0b002a5 with merge base f4b01a8 (image):
NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
  • pull / android / build-android (gh) (trunk failure)
    ImportError: /opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/executorch/extension/training/pybindings/_training_lib.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE
This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Jul 9, 2026
 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Jul 9, 2026
@JCNTHJCNTH
mentioned this pull request
Jul 9, 2026
 Merged

@github-actions
github-actionsBot
commented
Jul 9, 2026

Copy link


Copy Markdown

This PR needs a release notes: label
If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.
To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"
For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.
@JCNTH
Update
1f9198a
[ghstack-poisoned]
@JCNTH
Update
8030226
[ghstack-poisoned]
SS-JIA
SS-JIA
requested changes
Jul 9, 2026
View reviewed changes
@SS-JIASS-JIA
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.
SS-JIA
SS-JIA
approved these changes
Jul 9, 2026
View reviewed changes
@SS-JIASS-JIA
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.
@JCNTH
Update
7465dc1
[ghstack-poisoned]
@JCNTH
Update
4af5d96
[ghstack-poisoned]
@JCNTH
Update
0b002a5
[ghstack-poisoned]
@meta-codesync
meta-codesyncBot
merged commit 22e1ef3
into
gh/JCNTH/15/baseJul 9, 2026

177 of 183 checks passed

@meta-codesync
meta-codesyncBot
deleted the

gh/JCNTH/15/head

branch
July 9, 2026 21:41
@pytorchbotpytorchbot
mentioned this pull request
Jul 9, 2026
 Merged

JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
63122c1
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
8c7ddb8
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e39a350
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
248a688
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
c677485
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
d9c9db0
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
0cc2d41
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e27ac6d
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
45ea327
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
cb131ce
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
953f569
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e2c4392
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@SS-JIASS-JIASS-JIA approved these changes
@larryliu0820larryliu0820Awaiting requested review from larryliu0820
@kirklandsignkirklandsignAwaiting requested review from kirklandsign

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.meta-exported

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@JCNTH@SS-JIA@nil-is-all

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill GEMM - #20800
Merged
meta-codesync[bot] merged 6 commits into
gh/JCNTH/15/basefrom 
gh/JCNTH/15/head
Jul 9, 2026
Merged
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill GEMM#20800
meta-codesync[bot] merged 6 commits into
gh/JCNTH/15/basefrom 
gh/JCNTH/15/head
Conversation
@JCNTH
@JCNTHJCNTH
commented
Jul 9, 2026•

edited

Loading

Copy link


Copy Markdown

Contributor
Stack from ghstack (oldest at bottom):
+46-56% end-to-end prefill tok/s over the shipped f16 steel q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).
Problem: the f16 steel prefill GEMM (and its packed-word-dequant variant pwdq) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
Solution: an f16-accumulate variant of the pwdq kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with fma() (mirrors MLC's array<f16,16> reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (pwdq): f16 multiply, f32 accumulate.

After (pwdqf16acc): f16 multiply, f16 accumulate, f32 epilogue.
Implementation:
  • New ACC=half fork in the shared q4gsw_linear_gemm_steel.wgsl template (a shader_variants entry in q4gsw_linear_gemm_steel.yaml generates q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h) — no standalone shader file.
  • Opt-in via the enable_f16_accumulate_gemm runtime spec (a load-time BackendOption read in WebGPUBackend::init, threaded through WebGPUGraph::build(..., f16_accumulate_gemm) -> graph.f16_accumulate_gemm()), default OFF — no CMake option or compile flag.
  • When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and group_size % BK == 0; else the f32-accumulate pwdq / half / f32 kernels run (fail-closed).
Constraints: LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.

@exported-using-ghexport
Differential Revision: D111163606
Differential Revision: D111163606
@JCNTH
Update
d29f101
[ghstack-poisoned]
@pytorch-bot
pytorch-botBot
commented
Jul 9, 2026•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20800
Note: Links to docs will display an error until the docs builds have been completed.
❌ 3 New Failures, 12 Pending, 1 Unrelated Failure
As of commit 0b002a5 with merge base f4b01a8 (image):
NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
  • pull / android / build-android (gh) (trunk failure)
    ImportError: /opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/executorch/extension/training/pybindings/_training_lib.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE
This comment was automatically generated by Dr. CI and updates every 15 minutes.

This was referenced Jul 9, 2026
 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

 Merged

@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Jul 9, 2026
@JCNTHJCNTH
mentioned this pull request
Jul 9, 2026
 Merged

@github-actions
github-actionsBot
commented
Jul 9, 2026

Copy link


Copy Markdown

This PR needs a release notes: label
If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.
To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"
For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.
@JCNTH
Update
1f9198a
[ghstack-poisoned]
@JCNTH
Update
8030226
[ghstack-poisoned]
SS-JIA
SS-JIA
requested changes
Jul 9, 2026
View reviewed changes
@SS-JIASS-JIA
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.
SS-JIA
SS-JIA
approved these changes
Jul 9, 2026
View reviewed changes
@SS-JIASS-JIA
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review automatically exported from Phabricator review in Meta.
@JCNTH
Update
7465dc1
[ghstack-poisoned]
@JCNTH
Update
4af5d96
[ghstack-poisoned]
@JCNTH
Update
0b002a5
[ghstack-poisoned]
@meta-codesync
meta-codesyncBot
merged commit 22e1ef3
into
gh/JCNTH/15/baseJul 9, 2026

177 of 183 checks passed

@meta-codesync
meta-codesyncBot
deleted the

gh/JCNTH/15/head

branch
July 9, 2026 21:41
@pytorchbotpytorchbot
mentioned this pull request
Jul 9, 2026
 Merged

JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
63122c1
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
8c7ddb8
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e39a350
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
248a688
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
c677485
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
d9c9db0
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
0cc2d41
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e27ac6d
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
45ea327
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
cb131ce
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
953f569
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
JCNTH added a commit
that referenced
this pull request
Jul 9, 2026
@JCNTH
[ExecuTorch][WebGPU] f16-accumulate (pwdqf16acc) steel q4gsw prefill …
e2c4392
…GEMM
Pull Request resolved: #20800
**+46-56% end-to-end prefill tok/s over the shipped f16 `steel` q4gsw GEMM (Apple M4 Pro / Chrome Canary), behind an opt-in runtime spec (default OFF); perplexity held (13.32 -> 13.37, +0.05).**
**Problem:** the f16 `steel` prefill GEMM (and its packed-word-dequant variant `pwdq`) accumulates its 4x4 register tile in f32. WebLLM/MLC accumulate in f16, which halves the accumulator footprint and raises occupancy — the largest remaining prefill gap vs MLC on Apple.
**Solution:** an f16-accumulate variant of the `pwdq` kernel — identical staging (dequant-once + hoisted scale) and 64x64/256-thread/BK=16 geometry, but the 4x4 accumulator is kept in f16 with `fma()` (mirrors MLC's `array<f16,16>` reduction) and cast to f32 in the epilogue for the f32 output/bias.
Before (`pwdq`): f16 multiply, f32 accumulate.
After (`pwdqf16acc`): f16 multiply, f16 accumulate, f32 epilogue.
**Implementation:**
- New `ACC=half` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_f16acc_wgsl.h`) — no standalone shader file.
- Opt-in via the `enable_f16_accumulate_gemm` runtime spec (a load-time `BackendOption` read in `WebGPUBackend::init`, threaded through `WebGPUGraph::build(..., f16_accumulate_gemm)` -> `graph.f16_accumulate_gemm()`), default OFF — no CMake option or compile flag.
- When the spec is set, overrides the f32-accumulate steel kernels for M>1 prefill whenever the device negotiated shader-f16 and `group_size % BK == 0`; else the f32-accumulate `pwdq` / `half` / f32 kernels run (fail-closed).
**Constraints:** LOSSY — f16 accumulation over the full K (up to 8192) is not bit-exact, so it ships on a perplexity bar, not a bit-exact gate (as MLC does): measured Llama-3.2-1B int4 perplexity 13.32 -> 13.37 (+0.05) on the real prefill path. Default-OFF keeps upstream builds on the strict f32-accumulate golden; the runtime spec is opt-in for latency-sensitive deployments.
Co-authored-with: Claude Code.
ghstack-source-id: 401564900
@exported-using-ghexport
Differential Revision: [D111163606](https://our.internmc.facebook.com/intern/diff/D111163606/)
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@SS-JIASS-JIASS-JIA approved these changes
@larryliu0820larryliu0820Awaiting requested review from larryliu0820
@kirklandsignkirklandsignAwaiting requested review from kirklandsign

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.meta-exported

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@JCNTH@SS-JIA@nil-is-all