[ExecuTorch][WebGPU] Packed-word-dequant f16 steel q4gsw prefill GEMM by JCNTH · Pull Request #20798 · pytorch/executorch

JCNTH · 2026-07-09T03:44:41Z

Stack from ghstack (oldest at bottom):

+24-28% microbench GFLOP/s (+11-16% end-to-end prefill tok/s) on the f16 steel q4gsw prefill GEMM, Apple M4 Pro / Chrome Canary — bit-exact.

Problem: the f16 steel prefill GEMM (the half variant) dequantizes weights per-nibble — each of the 256 threads extracts ONE 4-bit weight from a full u32 word, so every packed word is re-loaded ~8x and the per-column scale is re-read ~16x across the BK=16 tile. That redundant global traffic dominates the kernel on Apple.

Solution: a packed-word-dequant staging variant, bit-exact to the half kernel — identical As staging, compute loop, epilogue, and 64x64 tile / 256-thread / BK=16 geometry; only the weight-dequant staging differs.

Before (half): each thread extracts 1 nibble from a re-loaded word; scale re-read per nibble.
After (pwdq): threads [0,BN) each stage one full BK-column — load the column's two u32 words ONCE, unpack all 16 nibbles, read the per-column scale ONCE.

Implementation:

New PWDQ fork in the shared q4gsw_linear_gemm_steel.wgsl template (a shader_variants entry in q4gsw_linear_gemm_steel.yaml generates q4gsw_linear_gemm_steel_half_pwdq_wgsl.h) — no standalone shader file. Selected at runtime when the device negotiated shader-f16 (ctx->shader_f16_supported) AND group_size % BK == 0 (the hoisted scale must be constant across the BK tile), else the per-nibble half kernel.
No new build option, dispatch, or bindings — same tile/count as steel; only the generated shader variant differs.

Constraints: requires K % BK == 0 (the steel route already guarantees it, making K_packed = K/2 a multiple of 8 so every column is u32-aligned) and group_size % BK == 0 (all real q4 group sizes: 32/64/128). f16-multiply / f32-accumulate, so numerically identical to the half kernel.

Co-authored-with: Claude Code.
@exported-using-ghexport

Differential Revision: D111163510

[ghstack-poisoned]

pytorch-bot · 2026-07-09T03:44:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20798

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 12 Pending, 1 Unrelated Failure

As of commit 06fe79f with merge base f4b01a8 ():

NEW FAILURES - The following jobs have failed:

pull / test-binary-size-linux-gcc / linux-job (gh)
RuntimeError: Command docker exec -t cdc170ae7490e2c1117b65df02e5cbab28363b8a504762a070b9407ba5063f51 /exec failed with exit code 1
pull / unittest / linux / linux-job (gh)
RuntimeError: Command docker exec -t 72358c6f4de215af738fd6e1fecdf8fc7d1de0d60e8a715f633d644049232ae4 /exec failed with exit code 1
pull / unittest-editable / linux / linux-job (gh)
RuntimeError: Command docker exec -t 8b82a8aadc5eec4e9f7ea6d8e7949eac0c5ca84b3aa86653409095c8bdcf872b /exec failed with exit code 1

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / android / build-android (gh) (trunk failure)
ImportError: /opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/executorch/extension/training/pybindings/_training_lib.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2026-07-09T03:45:43Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[ghstack-poisoned]

SS-JIA

Review automatically exported from Phabricator review in Meta.

[ghstack-poisoned]

Pull Request resolved: #20798 **+24-28% microbench GFLOP/s (+11-16% end-to-end prefill tok/s) on the f16 `steel` q4gsw prefill GEMM, Apple M4 Pro / Chrome Canary — bit-exact.** **Problem:** the f16 `steel` prefill GEMM (the `half` variant) dequantizes weights per-nibble — each of the 256 threads extracts ONE 4-bit weight from a full `u32` word, so every packed word is re-loaded ~8x and the per-column scale is re-read ~16x across the `BK=16` tile. That redundant global traffic dominates the kernel on Apple. **Solution:** a packed-word-dequant staging variant, bit-exact to the `half` kernel — identical `As` staging, compute loop, epilogue, and `64x64` tile / 256-thread / `BK=16` geometry; only the weight-dequant staging differs. Before (`half`): each thread extracts 1 nibble from a re-loaded word; scale re-read per nibble. After (`pwdq`): threads `[0,BN)` each stage one full `BK`-column — load the column's two `u32` words ONCE, unpack all 16 nibbles, read the per-column scale ONCE. **Implementation:** - New `PWDQ` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_wgsl.h`) — no standalone shader file. Selected at runtime when the device negotiated shader-f16 (`ctx->shader_f16_supported`) AND `group_size % BK == 0` (the hoisted scale must be constant across the `BK` tile), else the per-nibble `half` kernel. - No new build option, dispatch, or bindings — same tile/count as `steel`; only the generated shader variant differs. **Constraints:** requires `K % BK == 0` (the steel route already guarantees it, making `K_packed = K/2` a multiple of 8 so every column is `u32`-aligned) and `group_size % BK == 0` (all real q4 group sizes: 32/64/128). f16-multiply / f32-accumulate, so numerically identical to the `half` kernel. Co-authored-with: Claude Code. ghstack-source-id: 401564878 @exported-using-ghexport Differential Revision: [D111163510](https://our.internmc.facebook.com/intern/diff/D111163510/)

JCNTH · 2026-07-09T03:44:41Z

Stack from ghstack (oldest at bottom):

+24-28% microbench GFLOP/s (+11-16% end-to-end prefill tok/s) on the f16 steel q4gsw prefill GEMM, Apple M4 Pro / Chrome Canary — bit-exact.

Problem: the f16 steel prefill GEMM (the half variant) dequantizes weights per-nibble — each of the 256 threads extracts ONE 4-bit weight from a full u32 word, so every packed word is re-loaded ~8x and the per-column scale is re-read ~16x across the BK=16 tile. That redundant global traffic dominates the kernel on Apple.

Solution: a packed-word-dequant staging variant, bit-exact to the half kernel — identical As staging, compute loop, epilogue, and 64x64 tile / 256-thread / BK=16 geometry; only the weight-dequant staging differs.

Before (half): each thread extracts 1 nibble from a re-loaded word; scale re-read per nibble.
After (pwdq): threads [0,BN) each stage one full BK-column — load the column's two u32 words ONCE, unpack all 16 nibbles, read the per-column scale ONCE.

Implementation:

New PWDQ fork in the shared q4gsw_linear_gemm_steel.wgsl template (a shader_variants entry in q4gsw_linear_gemm_steel.yaml generates q4gsw_linear_gemm_steel_half_pwdq_wgsl.h) — no standalone shader file. Selected at runtime when the device negotiated shader-f16 (ctx->shader_f16_supported) AND group_size % BK == 0 (the hoisted scale must be constant across the BK tile), else the per-nibble half kernel.
No new build option, dispatch, or bindings — same tile/count as steel; only the generated shader variant differs.

Constraints: requires K % BK == 0 (the steel route already guarantees it, making K_packed = K/2 a multiple of 8 so every column is u32-aligned) and group_size % BK == 0 (all real q4 group sizes: 32/64/128). f16-multiply / f32-accumulate, so numerically identical to the half kernel.

Co-authored-with: Claude Code.
@exported-using-ghexport

Differential Revision: D111163510

[ghstack-poisoned]

pytorch-bot · 2026-07-09T03:44:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20798

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 12 Pending, 1 Unrelated Failure

As of commit 06fe79f with merge base f4b01a8 ():

NEW FAILURES - The following jobs have failed:

pull / test-binary-size-linux-gcc / linux-job (gh)
RuntimeError: Command docker exec -t cdc170ae7490e2c1117b65df02e5cbab28363b8a504762a070b9407ba5063f51 /exec failed with exit code 1
pull / unittest / linux / linux-job (gh)
RuntimeError: Command docker exec -t 72358c6f4de215af738fd6e1fecdf8fc7d1de0d60e8a715f633d644049232ae4 /exec failed with exit code 1
pull / unittest-editable / linux / linux-job (gh)
RuntimeError: Command docker exec -t 8b82a8aadc5eec4e9f7ea6d8e7949eac0c5ca84b3aa86653409095c8bdcf872b /exec failed with exit code 1

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / android / build-android (gh) (trunk failure)
ImportError: /opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/executorch/extension/training/pybindings/_training_lib.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2026-07-09T03:45:43Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[ghstack-poisoned]

SS-JIA

Review automatically exported from Phabricator review in Meta.

[ghstack-poisoned]

Pull Request resolved: #20798 **+24-28% microbench GFLOP/s (+11-16% end-to-end prefill tok/s) on the f16 `steel` q4gsw prefill GEMM, Apple M4 Pro / Chrome Canary — bit-exact.** **Problem:** the f16 `steel` prefill GEMM (the `half` variant) dequantizes weights per-nibble — each of the 256 threads extracts ONE 4-bit weight from a full `u32` word, so every packed word is re-loaded ~8x and the per-column scale is re-read ~16x across the `BK=16` tile. That redundant global traffic dominates the kernel on Apple. **Solution:** a packed-word-dequant staging variant, bit-exact to the `half` kernel — identical `As` staging, compute loop, epilogue, and `64x64` tile / 256-thread / `BK=16` geometry; only the weight-dequant staging differs. Before (`half`): each thread extracts 1 nibble from a re-loaded word; scale re-read per nibble. After (`pwdq`): threads `[0,BN)` each stage one full `BK`-column — load the column's two `u32` words ONCE, unpack all 16 nibbles, read the per-column scale ONCE. **Implementation:** - New `PWDQ` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_wgsl.h`) — no standalone shader file. Selected at runtime when the device negotiated shader-f16 (`ctx->shader_f16_supported`) AND `group_size % BK == 0` (the hoisted scale must be constant across the `BK` tile), else the per-nibble `half` kernel. - No new build option, dispatch, or bindings — same tile/count as `steel`; only the generated shader variant differs. **Constraints:** requires `K % BK == 0` (the steel route already guarantees it, making `K_packed = K/2` a multiple of 8 so every column is `u32`-aligned) and `group_size % BK == 0` (all real q4 group sizes: 32/64/128). f16-multiply / f32-accumulate, so numerically identical to the `half` kernel. Co-authored-with: Claude Code. ghstack-source-id: 401564878 @exported-using-ghexport Differential Revision: [D111163510](https://our.internmc.facebook.com/intern/diff/D111163510/)

JCNTH · 2026-07-09T03:44:41Z

Stack from ghstack (oldest at bottom):

+24-28% microbench GFLOP/s (+11-16% end-to-end prefill tok/s) on the f16 steel q4gsw prefill GEMM, Apple M4 Pro / Chrome Canary — bit-exact.

Problem: the f16 steel prefill GEMM (the half variant) dequantizes weights per-nibble — each of the 256 threads extracts ONE 4-bit weight from a full u32 word, so every packed word is re-loaded ~8x and the per-column scale is re-read ~16x across the BK=16 tile. That redundant global traffic dominates the kernel on Apple.

Solution: a packed-word-dequant staging variant, bit-exact to the half kernel — identical As staging, compute loop, epilogue, and 64x64 tile / 256-thread / BK=16 geometry; only the weight-dequant staging differs.

Before (half): each thread extracts 1 nibble from a re-loaded word; scale re-read per nibble.
After (pwdq): threads [0,BN) each stage one full BK-column — load the column's two u32 words ONCE, unpack all 16 nibbles, read the per-column scale ONCE.

Implementation:

New PWDQ fork in the shared q4gsw_linear_gemm_steel.wgsl template (a shader_variants entry in q4gsw_linear_gemm_steel.yaml generates q4gsw_linear_gemm_steel_half_pwdq_wgsl.h) — no standalone shader file. Selected at runtime when the device negotiated shader-f16 (ctx->shader_f16_supported) AND group_size % BK == 0 (the hoisted scale must be constant across the BK tile), else the per-nibble half kernel.
No new build option, dispatch, or bindings — same tile/count as steel; only the generated shader variant differs.

Constraints: requires K % BK == 0 (the steel route already guarantees it, making K_packed = K/2 a multiple of 8 so every column is u32-aligned) and group_size % BK == 0 (all real q4 group sizes: 32/64/128). f16-multiply / f32-accumulate, so numerically identical to the half kernel.

Co-authored-with: Claude Code.
@exported-using-ghexport

Differential Revision: D111163510

[ghstack-poisoned]

pytorch-bot · 2026-07-09T03:44:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20798

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 12 Pending, 1 Unrelated Failure

As of commit 06fe79f with merge base f4b01a8 ():

NEW FAILURES - The following jobs have failed:

pull / test-binary-size-linux-gcc / linux-job (gh)
RuntimeError: Command docker exec -t cdc170ae7490e2c1117b65df02e5cbab28363b8a504762a070b9407ba5063f51 /exec failed with exit code 1
pull / unittest / linux / linux-job (gh)
RuntimeError: Command docker exec -t 72358c6f4de215af738fd6e1fecdf8fc7d1de0d60e8a715f633d644049232ae4 /exec failed with exit code 1
pull / unittest-editable / linux / linux-job (gh)
RuntimeError: Command docker exec -t 8b82a8aadc5eec4e9f7ea6d8e7949eac0c5ca84b3aa86653409095c8bdcf872b /exec failed with exit code 1

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / android / build-android (gh) (trunk failure)
ImportError: /opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/executorch/extension/training/pybindings/_training_lib.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2026-07-09T03:45:43Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[ghstack-poisoned]

SS-JIA

Review automatically exported from Phabricator review in Meta.

[ghstack-poisoned]

Pull Request resolved: #20798 **+24-28% microbench GFLOP/s (+11-16% end-to-end prefill tok/s) on the f16 `steel` q4gsw prefill GEMM, Apple M4 Pro / Chrome Canary — bit-exact.** **Problem:** the f16 `steel` prefill GEMM (the `half` variant) dequantizes weights per-nibble — each of the 256 threads extracts ONE 4-bit weight from a full `u32` word, so every packed word is re-loaded ~8x and the per-column scale is re-read ~16x across the `BK=16` tile. That redundant global traffic dominates the kernel on Apple. **Solution:** a packed-word-dequant staging variant, bit-exact to the `half` kernel — identical `As` staging, compute loop, epilogue, and `64x64` tile / 256-thread / `BK=16` geometry; only the weight-dequant staging differs. Before (`half`): each thread extracts 1 nibble from a re-loaded word; scale re-read per nibble. After (`pwdq`): threads `[0,BN)` each stage one full `BK`-column — load the column's two `u32` words ONCE, unpack all 16 nibbles, read the per-column scale ONCE. **Implementation:** - New `PWDQ` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_wgsl.h`) — no standalone shader file. Selected at runtime when the device negotiated shader-f16 (`ctx->shader_f16_supported`) AND `group_size % BK == 0` (the hoisted scale must be constant across the `BK` tile), else the per-nibble `half` kernel. - No new build option, dispatch, or bindings — same tile/count as `steel`; only the generated shader variant differs. **Constraints:** requires `K % BK == 0` (the steel route already guarantees it, making `K_packed = K/2` a multiple of 8 so every column is `u32`-aligned) and `group_size % BK == 0` (all real q4 group sizes: 32/64/128). f16-multiply / f32-accumulate, so numerically identical to the `half` kernel. Co-authored-with: Claude Code. ghstack-source-id: 401564878 @exported-using-ghexport Differential Revision: [D111163510](https://our.internmc.facebook.com/intern/diff/D111163510/)

JCNTH · 2026-07-09T03:44:41Z

Stack from ghstack (oldest at bottom):

+24-28% microbench GFLOP/s (+11-16% end-to-end prefill tok/s) on the f16 steel q4gsw prefill GEMM, Apple M4 Pro / Chrome Canary — bit-exact.

Problem: the f16 steel prefill GEMM (the half variant) dequantizes weights per-nibble — each of the 256 threads extracts ONE 4-bit weight from a full u32 word, so every packed word is re-loaded ~8x and the per-column scale is re-read ~16x across the BK=16 tile. That redundant global traffic dominates the kernel on Apple.

Solution: a packed-word-dequant staging variant, bit-exact to the half kernel — identical As staging, compute loop, epilogue, and 64x64 tile / 256-thread / BK=16 geometry; only the weight-dequant staging differs.

Before (half): each thread extracts 1 nibble from a re-loaded word; scale re-read per nibble.
After (pwdq): threads [0,BN) each stage one full BK-column — load the column's two u32 words ONCE, unpack all 16 nibbles, read the per-column scale ONCE.

Implementation:

New PWDQ fork in the shared q4gsw_linear_gemm_steel.wgsl template (a shader_variants entry in q4gsw_linear_gemm_steel.yaml generates q4gsw_linear_gemm_steel_half_pwdq_wgsl.h) — no standalone shader file. Selected at runtime when the device negotiated shader-f16 (ctx->shader_f16_supported) AND group_size % BK == 0 (the hoisted scale must be constant across the BK tile), else the per-nibble half kernel.
No new build option, dispatch, or bindings — same tile/count as steel; only the generated shader variant differs.

Constraints: requires K % BK == 0 (the steel route already guarantees it, making K_packed = K/2 a multiple of 8 so every column is u32-aligned) and group_size % BK == 0 (all real q4 group sizes: 32/64/128). f16-multiply / f32-accumulate, so numerically identical to the half kernel.

Co-authored-with: Claude Code.
@exported-using-ghexport

Differential Revision: D111163510

[ghstack-poisoned]

pytorch-bot · 2026-07-09T03:44:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20798

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 12 Pending, 1 Unrelated Failure

As of commit 06fe79f with merge base f4b01a8 ():

NEW FAILURES - The following jobs have failed:

pull / test-binary-size-linux-gcc / linux-job (gh)
RuntimeError: Command docker exec -t cdc170ae7490e2c1117b65df02e5cbab28363b8a504762a070b9407ba5063f51 /exec failed with exit code 1
pull / unittest / linux / linux-job (gh)
RuntimeError: Command docker exec -t 72358c6f4de215af738fd6e1fecdf8fc7d1de0d60e8a715f633d644049232ae4 /exec failed with exit code 1
pull / unittest-editable / linux / linux-job (gh)
RuntimeError: Command docker exec -t 8b82a8aadc5eec4e9f7ea6d8e7949eac0c5ca84b3aa86653409095c8bdcf872b /exec failed with exit code 1

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / android / build-android (gh) (trunk failure)
ImportError: /opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/executorch/extension/training/pybindings/_training_lib.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2026-07-09T03:45:43Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[ghstack-poisoned]

SS-JIA

Review automatically exported from Phabricator review in Meta.

[ghstack-poisoned]

Pull Request resolved: #20798 **+24-28% microbench GFLOP/s (+11-16% end-to-end prefill tok/s) on the f16 `steel` q4gsw prefill GEMM, Apple M4 Pro / Chrome Canary — bit-exact.** **Problem:** the f16 `steel` prefill GEMM (the `half` variant) dequantizes weights per-nibble — each of the 256 threads extracts ONE 4-bit weight from a full `u32` word, so every packed word is re-loaded ~8x and the per-column scale is re-read ~16x across the `BK=16` tile. That redundant global traffic dominates the kernel on Apple. **Solution:** a packed-word-dequant staging variant, bit-exact to the `half` kernel — identical `As` staging, compute loop, epilogue, and `64x64` tile / 256-thread / `BK=16` geometry; only the weight-dequant staging differs. Before (`half`): each thread extracts 1 nibble from a re-loaded word; scale re-read per nibble. After (`pwdq`): threads `[0,BN)` each stage one full `BK`-column — load the column's two `u32` words ONCE, unpack all 16 nibbles, read the per-column scale ONCE. **Implementation:** - New `PWDQ` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_wgsl.h`) — no standalone shader file. Selected at runtime when the device negotiated shader-f16 (`ctx->shader_f16_supported`) AND `group_size % BK == 0` (the hoisted scale must be constant across the `BK` tile), else the per-nibble `half` kernel. - No new build option, dispatch, or bindings — same tile/count as `steel`; only the generated shader variant differs. **Constraints:** requires `K % BK == 0` (the steel route already guarantees it, making `K_packed = K/2` a multiple of 8 so every column is `u32`-aligned) and `group_size % BK == 0` (all real q4 group sizes: 32/64/128). f16-multiply / f32-accumulate, so numerically identical to the `half` kernel. Co-authored-with: Claude Code. ghstack-source-id: 401564878 @exported-using-ghexport Differential Revision: [D111163510](https://our.internmc.facebook.com/intern/diff/D111163510/)

JCNTH · 2026-07-09T03:44:41Z

Stack from ghstack (oldest at bottom):

+24-28% microbench GFLOP/s (+11-16% end-to-end prefill tok/s) on the f16 steel q4gsw prefill GEMM, Apple M4 Pro / Chrome Canary — bit-exact.

Problem: the f16 steel prefill GEMM (the half variant) dequantizes weights per-nibble — each of the 256 threads extracts ONE 4-bit weight from a full u32 word, so every packed word is re-loaded ~8x and the per-column scale is re-read ~16x across the BK=16 tile. That redundant global traffic dominates the kernel on Apple.

Solution: a packed-word-dequant staging variant, bit-exact to the half kernel — identical As staging, compute loop, epilogue, and 64x64 tile / 256-thread / BK=16 geometry; only the weight-dequant staging differs.

Before (half): each thread extracts 1 nibble from a re-loaded word; scale re-read per nibble.
After (pwdq): threads [0,BN) each stage one full BK-column — load the column's two u32 words ONCE, unpack all 16 nibbles, read the per-column scale ONCE.

Implementation:

New PWDQ fork in the shared q4gsw_linear_gemm_steel.wgsl template (a shader_variants entry in q4gsw_linear_gemm_steel.yaml generates q4gsw_linear_gemm_steel_half_pwdq_wgsl.h) — no standalone shader file. Selected at runtime when the device negotiated shader-f16 (ctx->shader_f16_supported) AND group_size % BK == 0 (the hoisted scale must be constant across the BK tile), else the per-nibble half kernel.
No new build option, dispatch, or bindings — same tile/count as steel; only the generated shader variant differs.

Constraints: requires K % BK == 0 (the steel route already guarantees it, making K_packed = K/2 a multiple of 8 so every column is u32-aligned) and group_size % BK == 0 (all real q4 group sizes: 32/64/128). f16-multiply / f32-accumulate, so numerically identical to the half kernel.

Co-authored-with: Claude Code.
@exported-using-ghexport

Differential Revision: D111163510

[ghstack-poisoned]

pytorch-bot · 2026-07-09T03:44:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20798

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 12 Pending, 1 Unrelated Failure

As of commit 06fe79f with merge base f4b01a8 ():

NEW FAILURES - The following jobs have failed:

pull / test-binary-size-linux-gcc / linux-job (gh)
RuntimeError: Command docker exec -t cdc170ae7490e2c1117b65df02e5cbab28363b8a504762a070b9407ba5063f51 /exec failed with exit code 1
pull / unittest / linux / linux-job (gh)
RuntimeError: Command docker exec -t 72358c6f4de215af738fd6e1fecdf8fc7d1de0d60e8a715f633d644049232ae4 /exec failed with exit code 1
pull / unittest-editable / linux / linux-job (gh)
RuntimeError: Command docker exec -t 8b82a8aadc5eec4e9f7ea6d8e7949eac0c5ca84b3aa86653409095c8bdcf872b /exec failed with exit code 1

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / android / build-android (gh) (trunk failure)
ImportError: /opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/executorch/extension/training/pybindings/_training_lib.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2026-07-09T03:45:43Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[ghstack-poisoned]

SS-JIA

Review automatically exported from Phabricator review in Meta.

[ghstack-poisoned]

Pull Request resolved: #20798 **+24-28% microbench GFLOP/s (+11-16% end-to-end prefill tok/s) on the f16 `steel` q4gsw prefill GEMM, Apple M4 Pro / Chrome Canary — bit-exact.** **Problem:** the f16 `steel` prefill GEMM (the `half` variant) dequantizes weights per-nibble — each of the 256 threads extracts ONE 4-bit weight from a full `u32` word, so every packed word is re-loaded ~8x and the per-column scale is re-read ~16x across the `BK=16` tile. That redundant global traffic dominates the kernel on Apple. **Solution:** a packed-word-dequant staging variant, bit-exact to the `half` kernel — identical `As` staging, compute loop, epilogue, and `64x64` tile / 256-thread / `BK=16` geometry; only the weight-dequant staging differs. Before (`half`): each thread extracts 1 nibble from a re-loaded word; scale re-read per nibble. After (`pwdq`): threads `[0,BN)` each stage one full `BK`-column — load the column's two `u32` words ONCE, unpack all 16 nibbles, read the per-column scale ONCE. **Implementation:** - New `PWDQ` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_wgsl.h`) — no standalone shader file. Selected at runtime when the device negotiated shader-f16 (`ctx->shader_f16_supported`) AND `group_size % BK == 0` (the hoisted scale must be constant across the `BK` tile), else the per-nibble `half` kernel. - No new build option, dispatch, or bindings — same tile/count as `steel`; only the generated shader variant differs. **Constraints:** requires `K % BK == 0` (the steel route already guarantees it, making `K_packed = K/2` a multiple of 8 so every column is `u32`-aligned) and `group_size % BK == 0` (all real q4 group sizes: 32/64/128). f16-multiply / f32-accumulate, so numerically identical to the `half` kernel. Co-authored-with: Claude Code. ghstack-source-id: 401564878 @exported-using-ghexport Differential Revision: [D111163510](https://our.internmc.facebook.com/intern/diff/D111163510/)

JCNTH · 2026-07-09T03:44:41Z

Stack from ghstack (oldest at bottom):

+24-28% microbench GFLOP/s (+11-16% end-to-end prefill tok/s) on the f16 steel q4gsw prefill GEMM, Apple M4 Pro / Chrome Canary — bit-exact.

Problem: the f16 steel prefill GEMM (the half variant) dequantizes weights per-nibble — each of the 256 threads extracts ONE 4-bit weight from a full u32 word, so every packed word is re-loaded ~8x and the per-column scale is re-read ~16x across the BK=16 tile. That redundant global traffic dominates the kernel on Apple.

Solution: a packed-word-dequant staging variant, bit-exact to the half kernel — identical As staging, compute loop, epilogue, and 64x64 tile / 256-thread / BK=16 geometry; only the weight-dequant staging differs.

Before (half): each thread extracts 1 nibble from a re-loaded word; scale re-read per nibble.
After (pwdq): threads [0,BN) each stage one full BK-column — load the column's two u32 words ONCE, unpack all 16 nibbles, read the per-column scale ONCE.

Implementation:

New PWDQ fork in the shared q4gsw_linear_gemm_steel.wgsl template (a shader_variants entry in q4gsw_linear_gemm_steel.yaml generates q4gsw_linear_gemm_steel_half_pwdq_wgsl.h) — no standalone shader file. Selected at runtime when the device negotiated shader-f16 (ctx->shader_f16_supported) AND group_size % BK == 0 (the hoisted scale must be constant across the BK tile), else the per-nibble half kernel.
No new build option, dispatch, or bindings — same tile/count as steel; only the generated shader variant differs.

Constraints: requires K % BK == 0 (the steel route already guarantees it, making K_packed = K/2 a multiple of 8 so every column is u32-aligned) and group_size % BK == 0 (all real q4 group sizes: 32/64/128). f16-multiply / f32-accumulate, so numerically identical to the half kernel.

Co-authored-with: Claude Code.
@exported-using-ghexport

Differential Revision: D111163510

[ghstack-poisoned]

pytorch-bot · 2026-07-09T03:44:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20798

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 12 Pending, 1 Unrelated Failure

As of commit 06fe79f with merge base f4b01a8 ():

NEW FAILURES - The following jobs have failed:

pull / test-binary-size-linux-gcc / linux-job (gh)
RuntimeError: Command docker exec -t cdc170ae7490e2c1117b65df02e5cbab28363b8a504762a070b9407ba5063f51 /exec failed with exit code 1
pull / unittest / linux / linux-job (gh)
RuntimeError: Command docker exec -t 72358c6f4de215af738fd6e1fecdf8fc7d1de0d60e8a715f633d644049232ae4 /exec failed with exit code 1
pull / unittest-editable / linux / linux-job (gh)
RuntimeError: Command docker exec -t 8b82a8aadc5eec4e9f7ea6d8e7949eac0c5ca84b3aa86653409095c8bdcf872b /exec failed with exit code 1

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / android / build-android (gh) (trunk failure)
ImportError: /opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/executorch/extension/training/pybindings/_training_lib.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2026-07-09T03:45:43Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[ghstack-poisoned]

SS-JIA

Review automatically exported from Phabricator review in Meta.

[ghstack-poisoned]

Pull Request resolved: #20798 **+24-28% microbench GFLOP/s (+11-16% end-to-end prefill tok/s) on the f16 `steel` q4gsw prefill GEMM, Apple M4 Pro / Chrome Canary — bit-exact.** **Problem:** the f16 `steel` prefill GEMM (the `half` variant) dequantizes weights per-nibble — each of the 256 threads extracts ONE 4-bit weight from a full `u32` word, so every packed word is re-loaded ~8x and the per-column scale is re-read ~16x across the `BK=16` tile. That redundant global traffic dominates the kernel on Apple. **Solution:** a packed-word-dequant staging variant, bit-exact to the `half` kernel — identical `As` staging, compute loop, epilogue, and `64x64` tile / 256-thread / `BK=16` geometry; only the weight-dequant staging differs. Before (`half`): each thread extracts 1 nibble from a re-loaded word; scale re-read per nibble. After (`pwdq`): threads `[0,BN)` each stage one full `BK`-column — load the column's two `u32` words ONCE, unpack all 16 nibbles, read the per-column scale ONCE. **Implementation:** - New `PWDQ` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_wgsl.h`) — no standalone shader file. Selected at runtime when the device negotiated shader-f16 (`ctx->shader_f16_supported`) AND `group_size % BK == 0` (the hoisted scale must be constant across the `BK` tile), else the per-nibble `half` kernel. - No new build option, dispatch, or bindings — same tile/count as `steel`; only the generated shader variant differs. **Constraints:** requires `K % BK == 0` (the steel route already guarantees it, making `K_packed = K/2` a multiple of 8 so every column is `u32`-aligned) and `group_size % BK == 0` (all real q4 group sizes: 32/64/128). f16-multiply / f32-accumulate, so numerically identical to the `half` kernel. Co-authored-with: Claude Code. ghstack-source-id: 401564878 @exported-using-ghexport Differential Revision: [D111163510](https://our.internmc.facebook.com/intern/diff/D111163510/)

JCNTH · 2026-07-09T03:44:41Z

Stack from ghstack (oldest at bottom):

+24-28% microbench GFLOP/s (+11-16% end-to-end prefill tok/s) on the f16 steel q4gsw prefill GEMM, Apple M4 Pro / Chrome Canary — bit-exact.

Problem: the f16 steel prefill GEMM (the half variant) dequantizes weights per-nibble — each of the 256 threads extracts ONE 4-bit weight from a full u32 word, so every packed word is re-loaded ~8x and the per-column scale is re-read ~16x across the BK=16 tile. That redundant global traffic dominates the kernel on Apple.

Solution: a packed-word-dequant staging variant, bit-exact to the half kernel — identical As staging, compute loop, epilogue, and 64x64 tile / 256-thread / BK=16 geometry; only the weight-dequant staging differs.

Before (half): each thread extracts 1 nibble from a re-loaded word; scale re-read per nibble.
After (pwdq): threads [0,BN) each stage one full BK-column — load the column's two u32 words ONCE, unpack all 16 nibbles, read the per-column scale ONCE.

Implementation:

New PWDQ fork in the shared q4gsw_linear_gemm_steel.wgsl template (a shader_variants entry in q4gsw_linear_gemm_steel.yaml generates q4gsw_linear_gemm_steel_half_pwdq_wgsl.h) — no standalone shader file. Selected at runtime when the device negotiated shader-f16 (ctx->shader_f16_supported) AND group_size % BK == 0 (the hoisted scale must be constant across the BK tile), else the per-nibble half kernel.
No new build option, dispatch, or bindings — same tile/count as steel; only the generated shader variant differs.

Constraints: requires K % BK == 0 (the steel route already guarantees it, making K_packed = K/2 a multiple of 8 so every column is u32-aligned) and group_size % BK == 0 (all real q4 group sizes: 32/64/128). f16-multiply / f32-accumulate, so numerically identical to the half kernel.

Co-authored-with: Claude Code.
@exported-using-ghexport

Differential Revision: D111163510

[ghstack-poisoned]

pytorch-bot · 2026-07-09T03:44:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20798

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 12 Pending, 1 Unrelated Failure

As of commit 06fe79f with merge base f4b01a8 ():

NEW FAILURES - The following jobs have failed:

pull / test-binary-size-linux-gcc / linux-job (gh)
RuntimeError: Command docker exec -t cdc170ae7490e2c1117b65df02e5cbab28363b8a504762a070b9407ba5063f51 /exec failed with exit code 1
pull / unittest / linux / linux-job (gh)
RuntimeError: Command docker exec -t 72358c6f4de215af738fd6e1fecdf8fc7d1de0d60e8a715f633d644049232ae4 /exec failed with exit code 1
pull / unittest-editable / linux / linux-job (gh)
RuntimeError: Command docker exec -t 8b82a8aadc5eec4e9f7ea6d8e7949eac0c5ca84b3aa86653409095c8bdcf872b /exec failed with exit code 1

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / android / build-android (gh) (trunk failure)
ImportError: /opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/executorch/extension/training/pybindings/_training_lib.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2026-07-09T03:45:43Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[ghstack-poisoned]

SS-JIA

Review automatically exported from Phabricator review in Meta.

[ghstack-poisoned]

Pull Request resolved: #20798 **+24-28% microbench GFLOP/s (+11-16% end-to-end prefill tok/s) on the f16 `steel` q4gsw prefill GEMM, Apple M4 Pro / Chrome Canary — bit-exact.** **Problem:** the f16 `steel` prefill GEMM (the `half` variant) dequantizes weights per-nibble — each of the 256 threads extracts ONE 4-bit weight from a full `u32` word, so every packed word is re-loaded ~8x and the per-column scale is re-read ~16x across the `BK=16` tile. That redundant global traffic dominates the kernel on Apple. **Solution:** a packed-word-dequant staging variant, bit-exact to the `half` kernel — identical `As` staging, compute loop, epilogue, and `64x64` tile / 256-thread / `BK=16` geometry; only the weight-dequant staging differs. Before (`half`): each thread extracts 1 nibble from a re-loaded word; scale re-read per nibble. After (`pwdq`): threads `[0,BN)` each stage one full `BK`-column — load the column's two `u32` words ONCE, unpack all 16 nibbles, read the per-column scale ONCE. **Implementation:** - New `PWDQ` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_wgsl.h`) — no standalone shader file. Selected at runtime when the device negotiated shader-f16 (`ctx->shader_f16_supported`) AND `group_size % BK == 0` (the hoisted scale must be constant across the `BK` tile), else the per-nibble `half` kernel. - No new build option, dispatch, or bindings — same tile/count as `steel`; only the generated shader variant differs. **Constraints:** requires `K % BK == 0` (the steel route already guarantees it, making `K_packed = K/2` a multiple of 8 so every column is `u32`-aligned) and `group_size % BK == 0` (all real q4 group sizes: 32/64/128). f16-multiply / f32-accumulate, so numerically identical to the `half` kernel. Co-authored-with: Claude Code. ghstack-source-id: 401564878 @exported-using-ghexport Differential Revision: [D111163510](https://our.internmc.facebook.com/intern/diff/D111163510/)

JCNTH · 2026-07-09T03:44:41Z

Stack from ghstack (oldest at bottom):

+24-28% microbench GFLOP/s (+11-16% end-to-end prefill tok/s) on the f16 steel q4gsw prefill GEMM, Apple M4 Pro / Chrome Canary — bit-exact.

Problem: the f16 steel prefill GEMM (the half variant) dequantizes weights per-nibble — each of the 256 threads extracts ONE 4-bit weight from a full u32 word, so every packed word is re-loaded ~8x and the per-column scale is re-read ~16x across the BK=16 tile. That redundant global traffic dominates the kernel on Apple.

Solution: a packed-word-dequant staging variant, bit-exact to the half kernel — identical As staging, compute loop, epilogue, and 64x64 tile / 256-thread / BK=16 geometry; only the weight-dequant staging differs.

Before (half): each thread extracts 1 nibble from a re-loaded word; scale re-read per nibble.
After (pwdq): threads [0,BN) each stage one full BK-column — load the column's two u32 words ONCE, unpack all 16 nibbles, read the per-column scale ONCE.

Implementation:

New PWDQ fork in the shared q4gsw_linear_gemm_steel.wgsl template (a shader_variants entry in q4gsw_linear_gemm_steel.yaml generates q4gsw_linear_gemm_steel_half_pwdq_wgsl.h) — no standalone shader file. Selected at runtime when the device negotiated shader-f16 (ctx->shader_f16_supported) AND group_size % BK == 0 (the hoisted scale must be constant across the BK tile), else the per-nibble half kernel.
No new build option, dispatch, or bindings — same tile/count as steel; only the generated shader variant differs.

Constraints: requires K % BK == 0 (the steel route already guarantees it, making K_packed = K/2 a multiple of 8 so every column is u32-aligned) and group_size % BK == 0 (all real q4 group sizes: 32/64/128). f16-multiply / f32-accumulate, so numerically identical to the half kernel.

Co-authored-with: Claude Code.
@exported-using-ghexport

Differential Revision: D111163510

[ghstack-poisoned]

pytorch-bot · 2026-07-09T03:44:45Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20798

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 3 New Failures, 12 Pending, 1 Unrelated Failure

As of commit 06fe79f with merge base f4b01a8 ():

NEW FAILURES - The following jobs have failed:

pull / test-binary-size-linux-gcc / linux-job (gh)
RuntimeError: Command docker exec -t cdc170ae7490e2c1117b65df02e5cbab28363b8a504762a070b9407ba5063f51 /exec failed with exit code 1
pull / unittest / linux / linux-job (gh)
RuntimeError: Command docker exec -t 72358c6f4de215af738fd6e1fecdf8fc7d1de0d60e8a715f633d644049232ae4 /exec failed with exit code 1
pull / unittest-editable / linux / linux-job (gh)
RuntimeError: Command docker exec -t 8b82a8aadc5eec4e9f7ea6d8e7949eac0c5ca84b3aa86653409095c8bdcf872b /exec failed with exit code 1

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / android / build-android (gh) (trunk failure)
ImportError: /opt/hostedtoolcache/Python/3.11.15/x64/lib/python3.11/site-packages/executorch/extension/training/pybindings/_training_lib.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104impl3cow23materialize_cow_storageERNS_11StorageImplE

This comment was automatically generated by Dr. CI and updates every 15 minutes.

github-actions · 2026-07-09T03:45:43Z

This PR needs a `release notes:` label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[ghstack-poisoned]

SS-JIA

Review automatically exported from Phabricator review in Meta.

[ghstack-poisoned]

Pull Request resolved: #20798 **+24-28% microbench GFLOP/s (+11-16% end-to-end prefill tok/s) on the f16 `steel` q4gsw prefill GEMM, Apple M4 Pro / Chrome Canary — bit-exact.** **Problem:** the f16 `steel` prefill GEMM (the `half` variant) dequantizes weights per-nibble — each of the 256 threads extracts ONE 4-bit weight from a full `u32` word, so every packed word is re-loaded ~8x and the per-column scale is re-read ~16x across the `BK=16` tile. That redundant global traffic dominates the kernel on Apple. **Solution:** a packed-word-dequant staging variant, bit-exact to the `half` kernel — identical `As` staging, compute loop, epilogue, and `64x64` tile / 256-thread / `BK=16` geometry; only the weight-dequant staging differs. Before (`half`): each thread extracts 1 nibble from a re-loaded word; scale re-read per nibble. After (`pwdq`): threads `[0,BN)` each stage one full `BK`-column — load the column's two `u32` words ONCE, unpack all 16 nibbles, read the per-column scale ONCE. **Implementation:** - New `PWDQ` fork in the shared `q4gsw_linear_gemm_steel.wgsl` template (a `shader_variants` entry in `q4gsw_linear_gemm_steel.yaml` generates `q4gsw_linear_gemm_steel_half_pwdq_wgsl.h`) — no standalone shader file. Selected at runtime when the device negotiated shader-f16 (`ctx->shader_f16_supported`) AND `group_size % BK == 0` (the hoisted scale must be constant across the `BK` tile), else the per-nibble `half` kernel. - No new build option, dispatch, or bindings — same tile/count as `steel`; only the generated shader variant differs. **Constraints:** requires `K % BK == 0` (the steel route already guarantees it, making `K_packed = K/2` a multiple of 8 so every column is `u32`-aligned) and `group_size % BK == 0` (all real q4 group sizes: 32/64/128). f16-multiply / f32-accumulate, so numerically identical to the `half` kernel. Co-authored-with: Claude Code. ghstack-source-id: 401564878 @exported-using-ghexport Differential Revision: [D111163510](https://our.internmc.facebook.com/intern/diff/D111163510/)

Update
42c7fe2
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 03:44 — with GitHub Actions Inactive

meta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 9, 2026

JCNTH temporarily deployed to cadence July 9, 2026 04:12 — with GitHub Actions Inactive

Update
43373b5
[ghstack-poisoned]

JCNTH had a problem deploying to cadence July 9, 2026 17:41 — with GitHub Actions Error

JCNTH requested a review from psiddh July 9, 2026 17:42

psiddh approved these changes Jul 9, 2026
View reviewed changes

Update
3d642d0
[ghstack-poisoned]

SS-JIA approved these changes Jul 9, 2026
View reviewed changes

meta-codesyncBot added the meta-exported label Jul 9, 2026

JCNTH temporarily deployed to cadence July 9, 2026 17:54 — with GitHub Actions Inactive

JCNTH had a problem deploying to cadence July 9, 2026 18:26 — with GitHub Actions Error

Update
8940b30
[ghstack-poisoned]

JCNTH had a problem deploying to cadence July 9, 2026 18:36 — with GitHub Actions Error

Update
cba2aa9
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 18:59 — with GitHub Actions Inactive

JCNTH temporarily deployed to cadence July 9, 2026 19:29 — with GitHub Actions Inactive

Update
06fe79f
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 20:30 — with GitHub Actions Inactive

JCNTH temporarily deployed to cadence July 9, 2026 20:58 — with GitHub Actions Inactive

meta-codesyncBot merged commit 6676cfa into gh/JCNTH/13/baseJul 9, 2026
177 of 183 checks passed

meta-codesyncBot deleted the gh/JCNTH/13/head branch July 9, 2026 21:40

pytorchbot mentioned this pull request Jul 9, 2026
[ExecuTorch][WebGPU] Packed-word-dequant f16 steel q4gsw prefill GEMM #20823
Merged

meta-codesyncBottemporarily deployed to cherry-pick-bot July 9, 2026 21:40 Inactive

Update
42c7fe2
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 03:44 — with GitHub Actions Inactive

meta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 9, 2026

JCNTH temporarily deployed to cadence July 9, 2026 04:12 — with GitHub Actions Inactive

Update
43373b5
[ghstack-poisoned]

JCNTH had a problem deploying to cadence July 9, 2026 17:41 — with GitHub Actions Error

JCNTH requested a review from psiddh July 9, 2026 17:42

psiddh approved these changes Jul 9, 2026
View reviewed changes

Update
3d642d0
[ghstack-poisoned]

SS-JIA approved these changes Jul 9, 2026
View reviewed changes

meta-codesyncBot added the meta-exported label Jul 9, 2026

JCNTH temporarily deployed to cadence July 9, 2026 17:54 — with GitHub Actions Inactive

JCNTH had a problem deploying to cadence July 9, 2026 18:26 — with GitHub Actions Error

Update
8940b30
[ghstack-poisoned]

JCNTH had a problem deploying to cadence July 9, 2026 18:36 — with GitHub Actions Error

Update
cba2aa9
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 18:59 — with GitHub Actions Inactive

JCNTH temporarily deployed to cadence July 9, 2026 19:29 — with GitHub Actions Inactive

Update
06fe79f
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 20:30 — with GitHub Actions Inactive

JCNTH temporarily deployed to cadence July 9, 2026 20:58 — with GitHub Actions Inactive

meta-codesyncBot merged commit 6676cfa into gh/JCNTH/13/baseJul 9, 2026
177 of 183 checks passed

meta-codesyncBot deleted the gh/JCNTH/13/head branch July 9, 2026 21:40

pytorchbot mentioned this pull request Jul 9, 2026
[ExecuTorch][WebGPU] Packed-word-dequant f16 steel q4gsw prefill GEMM #20823
Merged

meta-codesyncBottemporarily deployed to cherry-pick-bot July 9, 2026 21:40 Inactive

Update
42c7fe2
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 03:44 — with GitHub Actions Inactive

meta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 9, 2026

JCNTH temporarily deployed to cadence July 9, 2026 04:12 — with GitHub Actions Inactive

Update
43373b5
[ghstack-poisoned]

JCNTH had a problem deploying to cadence July 9, 2026 17:41 — with GitHub Actions Error

JCNTH requested a review from psiddh July 9, 2026 17:42

psiddh approved these changes Jul 9, 2026
View reviewed changes

Update
3d642d0
[ghstack-poisoned]

SS-JIA approved these changes Jul 9, 2026
View reviewed changes

meta-codesyncBot added the meta-exported label Jul 9, 2026

JCNTH temporarily deployed to cadence July 9, 2026 17:54 — with GitHub Actions Inactive

JCNTH had a problem deploying to cadence July 9, 2026 18:26 — with GitHub Actions Error

Update
8940b30
[ghstack-poisoned]

JCNTH had a problem deploying to cadence July 9, 2026 18:36 — with GitHub Actions Error

Update
cba2aa9
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 18:59 — with GitHub Actions Inactive

JCNTH temporarily deployed to cadence July 9, 2026 19:29 — with GitHub Actions Inactive

Update
06fe79f
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 20:30 — with GitHub Actions Inactive

JCNTH temporarily deployed to cadence July 9, 2026 20:58 — with GitHub Actions Inactive

meta-codesyncBot merged commit 6676cfa into gh/JCNTH/13/baseJul 9, 2026
177 of 183 checks passed

meta-codesyncBot deleted the gh/JCNTH/13/head branch July 9, 2026 21:40

pytorchbot mentioned this pull request Jul 9, 2026
[ExecuTorch][WebGPU] Packed-word-dequant f16 steel q4gsw prefill GEMM #20823
Merged

meta-codesyncBottemporarily deployed to cherry-pick-bot July 9, 2026 21:40 Inactive

Update
42c7fe2
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 03:44 — with GitHub Actions Inactive

meta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 9, 2026

JCNTH temporarily deployed to cadence July 9, 2026 04:12 — with GitHub Actions Inactive

Update
43373b5
[ghstack-poisoned]

JCNTH had a problem deploying to cadence July 9, 2026 17:41 — with GitHub Actions Error

JCNTH requested a review from psiddh July 9, 2026 17:42

psiddh approved these changes Jul 9, 2026
View reviewed changes

Update
3d642d0
[ghstack-poisoned]

SS-JIA approved these changes Jul 9, 2026
View reviewed changes

meta-codesyncBot added the meta-exported label Jul 9, 2026

JCNTH temporarily deployed to cadence July 9, 2026 17:54 — with GitHub Actions Inactive

JCNTH had a problem deploying to cadence July 9, 2026 18:26 — with GitHub Actions Error

Update
8940b30
[ghstack-poisoned]

JCNTH had a problem deploying to cadence July 9, 2026 18:36 — with GitHub Actions Error

Update
cba2aa9
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 18:59 — with GitHub Actions Inactive

JCNTH temporarily deployed to cadence July 9, 2026 19:29 — with GitHub Actions Inactive

Update
06fe79f
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 20:30 — with GitHub Actions Inactive

JCNTH temporarily deployed to cadence July 9, 2026 20:58 — with GitHub Actions Inactive

meta-codesyncBot merged commit 6676cfa into gh/JCNTH/13/baseJul 9, 2026
177 of 183 checks passed

meta-codesyncBot deleted the gh/JCNTH/13/head branch July 9, 2026 21:40

pytorchbot mentioned this pull request Jul 9, 2026
[ExecuTorch][WebGPU] Packed-word-dequant f16 steel q4gsw prefill GEMM #20823
Merged

meta-codesyncBottemporarily deployed to cherry-pick-bot July 9, 2026 21:40 Inactive

Update
42c7fe2
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 03:44 — with GitHub Actions Inactive

meta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 9, 2026

JCNTH temporarily deployed to cadence July 9, 2026 04:12 — with GitHub Actions Inactive

Update
43373b5
[ghstack-poisoned]

JCNTH had a problem deploying to cadence July 9, 2026 17:41 — with GitHub Actions Error

JCNTH requested a review from psiddh July 9, 2026 17:42

psiddh approved these changes Jul 9, 2026
View reviewed changes

Update
3d642d0
[ghstack-poisoned]

SS-JIA approved these changes Jul 9, 2026
View reviewed changes

meta-codesyncBot added the meta-exported label Jul 9, 2026

JCNTH temporarily deployed to cadence July 9, 2026 17:54 — with GitHub Actions Inactive

JCNTH had a problem deploying to cadence July 9, 2026 18:26 — with GitHub Actions Error

Update
8940b30
[ghstack-poisoned]

JCNTH had a problem deploying to cadence July 9, 2026 18:36 — with GitHub Actions Error

Update
cba2aa9
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 18:59 — with GitHub Actions Inactive

JCNTH temporarily deployed to cadence July 9, 2026 19:29 — with GitHub Actions Inactive

Update
06fe79f
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 20:30 — with GitHub Actions Inactive

JCNTH temporarily deployed to cadence July 9, 2026 20:58 — with GitHub Actions Inactive

meta-codesyncBot merged commit 6676cfa into gh/JCNTH/13/baseJul 9, 2026
177 of 183 checks passed

meta-codesyncBot deleted the gh/JCNTH/13/head branch July 9, 2026 21:40

pytorchbot mentioned this pull request Jul 9, 2026
[ExecuTorch][WebGPU] Packed-word-dequant f16 steel q4gsw prefill GEMM #20823
Merged

meta-codesyncBottemporarily deployed to cherry-pick-bot July 9, 2026 21:40 Inactive

Update
42c7fe2
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 03:44 — with GitHub Actions Inactive

meta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 9, 2026

JCNTH temporarily deployed to cadence July 9, 2026 04:12 — with GitHub Actions Inactive

Update
43373b5
[ghstack-poisoned]

JCNTH had a problem deploying to cadence July 9, 2026 17:41 — with GitHub Actions Error

JCNTH requested a review from psiddh July 9, 2026 17:42

psiddh approved these changes Jul 9, 2026
View reviewed changes

Update
3d642d0
[ghstack-poisoned]

SS-JIA approved these changes Jul 9, 2026
View reviewed changes

meta-codesyncBot added the meta-exported label Jul 9, 2026

JCNTH temporarily deployed to cadence July 9, 2026 17:54 — with GitHub Actions Inactive

JCNTH had a problem deploying to cadence July 9, 2026 18:26 — with GitHub Actions Error

Update
8940b30
[ghstack-poisoned]

JCNTH had a problem deploying to cadence July 9, 2026 18:36 — with GitHub Actions Error

Update
cba2aa9
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 18:59 — with GitHub Actions Inactive

JCNTH temporarily deployed to cadence July 9, 2026 19:29 — with GitHub Actions Inactive

Update
06fe79f
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 20:30 — with GitHub Actions Inactive

JCNTH temporarily deployed to cadence July 9, 2026 20:58 — with GitHub Actions Inactive

meta-codesyncBot merged commit 6676cfa into gh/JCNTH/13/baseJul 9, 2026
177 of 183 checks passed

meta-codesyncBot deleted the gh/JCNTH/13/head branch July 9, 2026 21:40

pytorchbot mentioned this pull request Jul 9, 2026
[ExecuTorch][WebGPU] Packed-word-dequant f16 steel q4gsw prefill GEMM #20823
Merged

meta-codesyncBottemporarily deployed to cherry-pick-bot July 9, 2026 21:40 Inactive

Update
42c7fe2
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 03:44 — with GitHub Actions Inactive

meta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 9, 2026

JCNTH temporarily deployed to cadence July 9, 2026 04:12 — with GitHub Actions Inactive

Update
43373b5
[ghstack-poisoned]

JCNTH had a problem deploying to cadence July 9, 2026 17:41 — with GitHub Actions Error

JCNTH requested a review from psiddh July 9, 2026 17:42

psiddh approved these changes Jul 9, 2026
View reviewed changes

Update
3d642d0
[ghstack-poisoned]

SS-JIA approved these changes Jul 9, 2026
View reviewed changes

meta-codesyncBot added the meta-exported label Jul 9, 2026

JCNTH temporarily deployed to cadence July 9, 2026 17:54 — with GitHub Actions Inactive

JCNTH had a problem deploying to cadence July 9, 2026 18:26 — with GitHub Actions Error

Update
8940b30
[ghstack-poisoned]

JCNTH had a problem deploying to cadence July 9, 2026 18:36 — with GitHub Actions Error

Update
cba2aa9
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 18:59 — with GitHub Actions Inactive

JCNTH temporarily deployed to cadence July 9, 2026 19:29 — with GitHub Actions Inactive

Update
06fe79f
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 20:30 — with GitHub Actions Inactive

JCNTH temporarily deployed to cadence July 9, 2026 20:58 — with GitHub Actions Inactive

meta-codesyncBot merged commit 6676cfa into gh/JCNTH/13/baseJul 9, 2026
177 of 183 checks passed

meta-codesyncBot deleted the gh/JCNTH/13/head branch July 9, 2026 21:40

pytorchbot mentioned this pull request Jul 9, 2026
[ExecuTorch][WebGPU] Packed-word-dequant f16 steel q4gsw prefill GEMM #20823
Merged

meta-codesyncBottemporarily deployed to cherry-pick-bot July 9, 2026 21:40 Inactive

Update
42c7fe2
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 03:44 — with GitHub Actions Inactive

meta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jul 9, 2026

JCNTH temporarily deployed to cadence July 9, 2026 04:12 — with GitHub Actions Inactive

Update
43373b5
[ghstack-poisoned]

JCNTH had a problem deploying to cadence July 9, 2026 17:41 — with GitHub Actions Error

JCNTH requested a review from psiddh July 9, 2026 17:42

psiddh approved these changes Jul 9, 2026
View reviewed changes

Update
3d642d0
[ghstack-poisoned]

SS-JIA approved these changes Jul 9, 2026
View reviewed changes

meta-codesyncBot added the meta-exported label Jul 9, 2026

JCNTH temporarily deployed to cadence July 9, 2026 17:54 — with GitHub Actions Inactive

JCNTH had a problem deploying to cadence July 9, 2026 18:26 — with GitHub Actions Error

Update
8940b30
[ghstack-poisoned]

JCNTH had a problem deploying to cadence July 9, 2026 18:36 — with GitHub Actions Error

Update
cba2aa9
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 18:59 — with GitHub Actions Inactive

JCNTH temporarily deployed to cadence July 9, 2026 19:29 — with GitHub Actions Inactive

Update
06fe79f
[ghstack-poisoned]

JCNTH temporarily deployed to cadence July 9, 2026 20:30 — with GitHub Actions Inactive

JCNTH temporarily deployed to cadence July 9, 2026 20:58 — with GitHub Actions Inactive

meta-codesyncBot merged commit 6676cfa into gh/JCNTH/13/baseJul 9, 2026
177 of 183 checks passed

meta-codesyncBot deleted the gh/JCNTH/13/head branch July 9, 2026 21:40

pytorchbot mentioned this pull request Jul 9, 2026
[ExecuTorch][WebGPU] Packed-word-dequant f16 steel q4gsw prefill GEMM #20823
Merged

meta-codesyncBottemporarily deployed to cherry-pick-bot July 9, 2026 21:40 Inactive

Uh oh!

Conversation

JCNTH commented Jul 9, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-botBot commented Jul 9, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20798

❌ 3 New Failures, 12 Pending, 1 Unrelated Failure

Uh oh!

github-actionsBot commented Jul 9, 2026

This PR needs a release notes: label

Uh oh!

SS-JIA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

JCNTH commented Jul 9, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-botBot commented Jul 9, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20798

❌ 3 New Failures, 12 Pending, 1 Unrelated Failure

Uh oh!

github-actionsBot commented Jul 9, 2026

This PR needs a release notes: label

Uh oh!

SS-JIA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

JCNTH commented Jul 9, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-botBot commented Jul 9, 2026• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/20798

❌ 3 New Failures, 12 Pending, 1 Unrelated Failure

Uh oh!

github-actionsBot commented Jul 9, 2026

This PR needs a release notes: label

Uh oh!

SS-JIA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

JCNTH commented Jul 9, 2026•
edited
Loading

pytorch-botBot commented Jul 9, 2026•
edited
Loading

This PR needs a `release notes:` label

JCNTH commented Jul 9, 2026•
edited
Loading

pytorch-botBot commented Jul 9, 2026•
edited
Loading

This PR needs a `release notes:` label

JCNTH commented Jul 9, 2026•
edited
Loading

pytorch-botBot commented Jul 9, 2026•
edited
Loading

This PR needs a `release notes:` label

JCNTH commented Jul 9, 2026•
edited
Loading

pytorch-botBot commented Jul 9, 2026•
edited
Loading

This PR needs a `release notes:` label

JCNTH commented Jul 9, 2026•
edited
Loading

pytorch-botBot commented Jul 9, 2026•
edited
Loading

This PR needs a `release notes:` label