Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w - #19632

Merged
psiddh merged 1 commit into
pytorch:mainfrom
CodeLinaro:dev1/quant/pcq2bit
Jun 8, 2026
Merged

Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w#19632
psiddh merged 1 commit into
pytorch:mainfrom
CodeLinaro:dev1/quant/pcq2bit

Conversation

@jethroqti

@jethroqtijethroqti commented May 18, 2026•
edited by pytorch-bot Bot
Loading

Copy link
Copy Markdown
Contributor

Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w

Summary:
1.Add 2-bits quantization basis 16a2w quantizer with standard symmetric
2.Support per channel and linear layers
3.Currently support soc model SM8850

Test plan:
python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_conv2d -b build-android -H ${HOST} -s ${SN} -m SM8850 python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_linear -b build-android -H ${HOST} -s ${SN} -m SM8850

cc @cccclai@cbilgin@abhinaykukkadapu

@pytorch-bot

pytorch-botBot commented May 18, 2026•
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19632

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Unrelated Failure

As of commit 83e6bba with merge base 02e57bd (image):

NEW FAILURE - The following job has failed:

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-clameta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label May 18, 2026
@jethroqti

jethroqti commented May 18, 2026

Copy link
Copy Markdown
ContributorAuthor

@pytorchbot label "release notes: qualcomm"

@pytorch-botpytorch-botBot added the release notes: qualcomm Changes to the Qualcomm backend delegate label May 18, 2026
@jethroqti

jethroqti commented May 18, 2026

Copy link
Copy Markdown
ContributorAuthor

This PR is used to support 2-bits quantization basis 16a2w. Please take a look. Thanks.
@psiddh@haowhsu-quic@shewu-quic@winskuo-quic@DannyYuyang-quic

@psiddh

psiddh commented May 28, 2026

Copy link
Copy Markdown
Contributor

@claude pls review

@claude

claudeBot commented May 28, 2026•
edited
Loading

Copy link
Copy Markdown

Claude finished @psiddh's task in 2m 53s —— View job

Code Review: 2-bits quantization 16a2w support

  • Read PR diff and changed files
  • Review backends/qualcomm/builders/node_visitor.py
  • Review backends/qualcomm/quantizer/qconfig.py
  • Review backends/qualcomm/quantizer/quantizer.py
  • Review backends/qualcomm/tests/test_qnn_delegate.py
  • Post review feedback

Overall the changes are well-structured and follow the existing patterns for 4-bit quantization. A few items to address:

1. make_qnn_per_tensor_config may misclassify 2-bit as 4-bit

File:backends/qualcomm/builders/node_visitor.py:278-287

The per-channel path correctly checks for <= 3 before <= 15, but make_qnn_per_tensor_config still has only the 4-bit check:

# special case for 4 bitsif (
quant_config[QCOM_DTYPE] ==torch.int8andquant_config[QCOM_QUANT_MAX] -quant_config[QCOM_QUANT_MIN] <=15
):
quant_config[QCOM_BITWIDTH] =4

Since 1 - (-2) = 3 <= 15, a 2-bit tensor that reaches this per-tensor path would be incorrectly classified as 4-bit with QCOM_BITWIDTH = 4. I understand that 2-bit only supports per-channel currently, but adding a defensive 2-bit check here (matching the per-channel pattern) would prevent silent misclassification if the path is hit unexpectedly.

Fix this →

2. Weight quantization range [-2, 1] — intentional asymmetry?

File:backends/qualcomm/quantizer/qconfig.py:242-244

The 2-bit weight uses quant_min=-2, quant_max=1 with per_tensor_symmetric. This is the full 2-bit signed range (4 values: {-2, -1, 0, 1}), which is asymmetric around zero. By contrast, the 4-bit config uses [-7, 7] — a truly symmetric range that sacrifices one code point.

For 2-bit, sacrificing one of only 4 code points (25% capacity) for true symmetry ([-1, 1]) is expensive, so using the full range makes sense as a practical tradeoff. Just confirming this is intentional and that the QNN runtime handles this correctly with per_tensor_symmetric qscheme — since the scheme name doesn't quite match the range.

3. Missing ch_axis in get_16a2w_qnn_ptq_config weight spec

File:backends/qualcomm/quantizer/qconfig.py:241-246

The comparable get_16a4w_qnn_ptq_config (line 285-291) includes ch_axis=0 in its weight QuantizationSpec, but get_16a2w_qnn_ptq_config omits it. Since the comment states "2 bits weight quantization only supports per channel and symmetric," should ch_axis=0 be included here for consistency? This config is used as the default (non-per-channel) entry in QUANT_CONFIG_DICT, so it may be hit for ops that don't get the per-channel annotator.

4. CI failure: test_qnn_backend_16a2w_conv2d

The CI shows this test is failing. Worth investigating whether this is an issue with the conv2d lowering path for 2-bit, or an environment/setup issue on the CI runner.

5. Minor: test loop variable

File:backends/qualcomm/tests/test_qnn_delegate.py:2486

fori, per_channel, use_biasin [
(1, True, False),
(2, True, True),
]:

The loop starts the index at 1 instead of 0. This is cosmetic (only affects subTest output), but using enumerate like the conv2d test would be more consistent:

fori, (per_channel, use_bias) inenumerate([
(True, False),
(True, True),
]):

Summary

The implementation correctly extends the existing quantization infrastructure for 2-bit support. The key concern is item #1 (potential misclassification in the per-tensor path) — the rest are minor or confirmatory. The CI failure in #4 should be resolved before merging.

@linux-foundation-easycla

linux-foundation-easyclaBot commented May 30, 2026•
edited
Loading

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

@jethroqti

jethroqti commented May 30, 2026

Copy link
Copy Markdown
ContributorAuthor

@pytorchbot label "release notes: qualcomm"

@psiddh

psiddh commented Jun 2, 2026

Copy link
Copy Markdown
Contributor

@claude Re review the latest fixes and changes

@claude

claudeBot commented Jun 2, 2026•
edited
Loading

Copy link
Copy Markdown

Claude finished @psiddh's task in 1m 28s —— View job

Re-review: 2-bits quantization 16a2w support (after fixes)

  • Read PR diff and latest changes
  • Review backends/qualcomm/builders/node_visitor.py
  • Review backends/qualcomm/quantizer/qconfig.py
  • Review backends/qualcomm/quantizer/quantizer.py
  • Review backends/qualcomm/quantizer/validators.py
  • Review backends/qualcomm/tests/test_qnn_delegate.py
  • Post review feedback

The fix commit (c3479b8) addresses the key issues raised in the first review. Here's the assessment of the current state:

Previously Raised Issues — Status

#IssueStatus
1Per-tensor misclassification of 2-bit as 4-bitFixed — assertion added in both node_visitor.py:278-282 and validators.py:286-290 to fail fast if 2-bit reaches per-tensor path
2Asymmetric range [-2, 1] with per_tensor_symmetricUnchanged — appears intentional for the hardware
3Missing ch_axis in get_16a2w_qnn_ptq_configNot addressed — acceptable since the assertion now guards this path
4CI test failureNeeds CI run to verify
5Test loop variableFixed — now uses enumerate() correctly

Current Code Assessment

The fixes look good. The assertion-based approach in the per-tensor paths is the right call — it's better to fail loudly than to silently misclassify.

One remaining minor concern in validators.py:306-309:

elifqspec.dtype==torch.int8andqspec.quant_max-qspec.quant_min<=3:
encoding_type= (
PyQnnManager.Qnn_QuantizationEncoding_t.QNN_QUANTIZATION_ENCODING_BW_AXIS_SCALE_OFFSET
)
elifqspec.dtype==torch.int8andqspec.quant_max-qspec.quant_min<=15:

This correctly handles 2-bit in the per-channel branch, but range_ was already computed earlier in the function (line 286). The per-channel branch recomputes qspec.quant_max - qspec.quant_min inline instead of using a local variable. This is purely cosmetic — not a bug.

Summary

The PR is in good shape after the fix commit. The core implementation correctly:

  • Extends per-channel config to handle torch.int2 with range [-2, 1]
  • Applies 2-bit masking (0x03) for tensor data encoding
  • Guards the per-tensor path with clear assertions
  • Properly registers use_16a2w in QUANT_CONFIG_DICT with the per-channel annotator
  • Tests both conv2d and linear with bias/no-bias variants

Remaining action item: CI approval needed to verify test_qnn_backend_16a2w_conv2d and test_qnn_backend_16a2w_linear pass on SM8850 hardware.

View job

@psiddh

psiddh commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

This has merge conflicts, Pls rebase and this is good to go

@psiddhpsiddh added the module: qnn Issues related to Qualcomm's QNN delegate and code under backends/qualcomm/ label Jun 4, 2026
@jethroqti
jethroqtiforce-pushed the dev1/quant/pcq2bit branch from c3479b8 to 4fddea0CompareJune 5, 2026 01:12
@jethroqti

jethroqti commented Jun 5, 2026

Copy link
Copy Markdown
ContributorAuthor

@pytorchbot label "release notes: qualcomm"

@jethroqti

jethroqti commented Jun 5, 2026

Copy link
Copy Markdown
ContributorAuthor

@psiddh

@psiddh

psiddh commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

@jethroqti can you check if this failure is related to the PR ? pull / test-qnn-delegate-linux / linux-job (pull_request)

@jethroqti
jethroqtiforce-pushed the dev1/quant/pcq2bit branch from 4fddea0 to e96a504CompareJune 5, 2026 10:36
@jethroqti

jethroqti commented Jun 5, 2026

Copy link
Copy Markdown
ContributorAuthor

@pytorchbot label "release notes: qualcomm"

@jethroqti

jethroqti commented Jun 6, 2026•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

@psiddh
If I use QNN SDK version >=2.41, I did not find the "test_qnn_backend_16a2w_conv2d - AssertionError: False is not true : ref_output".
Could you try to use newer version QNN_SDK 2.41.0.251128?
Because I found HTP fixed an issue in the quantizer in v2.40.

shewu-quic
shewu-quic reviewed Jun 8, 2026
View reviewed changes
Comment threadbackends/qualcomm/tests/test_qnn_delegate.py
shared_buffer=TestQNN.shared_buffer,
)

def test_qnn_backend_16a2w_conv2d(self):

@shewu-quicshewu-quicJun 8, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a skip condition based on qnn sdk version. Thanks

@jethroqti
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w
83e6bba
Summary:
1.Add 2-bits quantization basis 16a2w quantizer with standard symmetric
2.Support per channel and linear layers
3.Currently support soc model SM8850
Test plan:
python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_conv2d -b build-android -H ${HOST} -s ${SN} -m SM8850
python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_linear -b build-android -H ${HOST} -s ${SN} -m SM8850
@jethroqti
jethroqtiforce-pushed the dev1/quant/pcq2bit branch from e96a504 to 83e6bbaCompareJune 8, 2026 07:08
@jethroqti

jethroqti commented Jun 8, 2026

Copy link
Copy Markdown
ContributorAuthor

@pytorchbot label "release notes: qualcomm"

@psiddh
psiddh merged commit c4e3db0 into pytorch:mainJun 8, 2026
176 of 178 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@psiddhpsiddhpsiddh approved these changes
@abhinaykukkadapuabhinaykukkadapuAwaiting requested review from abhinaykukkadapuabhinaykukkadapu is a code owner
+1 more reviewer
@shewu-quicshewu-quicshewu-quic left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.module: qnnIssues related to Qualcomm's QNN delegate and code under backends/qualcomm/release notes: qualcommChanges to the Qualcomm backend delegate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jethroqti@psiddh@shewu-quic
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w - #19632
Merged
psiddh merged 1 commit into
pytorch:mainfrom 
CodeLinaro:dev1/quant/pcq2bit
Jun 8, 2026
Merged
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w#19632
psiddh merged 1 commit into
pytorch:mainfrom 
CodeLinaro:dev1/quant/pcq2bit
Conversation
@jethroqti
@jethroqtijethroqti
commented
May 18, 2026•

edited by pytorch-bot
Bot
Loading

Copy link


Copy Markdown

Contributor
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w
Summary:

1.Add 2-bits quantization basis 16a2w quantizer with standard symmetric

2.Support per channel and linear layers

3.Currently support soc model SM8850
Test plan:

python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_conv2d -b build-android -H ${HOST} -s ${SN} -m SM8850 python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_linear -b build-android -H ${HOST} -s ${SN} -m SM8850
cc @cccclai@cbilgin@abhinaykukkadapu
@pytorch-bot
pytorch-botBot
commented
May 18, 2026•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19632
Note: Links to docs will display an error until the docs builds have been completed.
❌ 1 New Failure, 1 Unrelated Failure
As of commit 83e6bba with merge base 02e57bd (image):
NEW FAILURE - The following job has failed:
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
May 18, 2026
@jethroqti
jethroqti
commented
May 18, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@pytorch-botpytorch-botBot
added
the 
release notes: qualcomm
Changes to the Qualcomm backend delegate
label
May 18, 2026
@jethroqti
jethroqti
commented
May 18, 2026

Copy link


Copy Markdown

ContributorAuthor
This PR is used to support 2-bits quantization basis 16a2w. Please take a look. Thanks.
@psiddh@haowhsu-quic@shewu-quic@winskuo-quic@DannyYuyang-quic
@psiddh
psiddh
commented
May 28, 2026

Copy link


Copy Markdown

Contributor
@claude pls review
@claude
claudeBot
commented
May 28, 2026•

edited

Loading

Copy link


Copy Markdown

Claude finished @psiddh's task in 2m 53s —— View job
Code Review: 2-bits quantization 16a2w support
  •  Read PR diff and changed files
  •  Review backends/qualcomm/builders/node_visitor.py
  •  Review backends/qualcomm/quantizer/qconfig.py
  •  Review backends/qualcomm/quantizer/quantizer.py
  •  Review backends/qualcomm/tests/test_qnn_delegate.py
  •  Post review feedback
Overall the changes are well-structured and follow the existing patterns for 4-bit quantization. A few items to address:
1. make_qnn_per_tensor_config may misclassify 2-bit as 4-bit
File:backends/qualcomm/builders/node_visitor.py:278-287
The per-channel path correctly checks for <= 3 before <= 15, but make_qnn_per_tensor_config still has only the 4-bit check:
# special case for 4 bitsif (
quant_config[QCOM_DTYPE] ==torch.int8andquant_config[QCOM_QUANT_MAX] -quant_config[QCOM_QUANT_MIN] <=15
):
quant_config[QCOM_BITWIDTH] =4
Since 1 - (-2) = 3 <= 15, a 2-bit tensor that reaches this per-tensor path would be incorrectly classified as 4-bit with QCOM_BITWIDTH = 4. I understand that 2-bit only supports per-channel currently, but adding a defensive 2-bit check here (matching the per-channel pattern) would prevent silent misclassification if the path is hit unexpectedly.
Fix this →
2. Weight quantization range [-2, 1] — intentional asymmetry?
File:backends/qualcomm/quantizer/qconfig.py:242-244
The 2-bit weight uses quant_min=-2, quant_max=1 with per_tensor_symmetric. This is the full 2-bit signed range (4 values: {-2, -1, 0, 1}), which is asymmetric around zero. By contrast, the 4-bit config uses [-7, 7] — a truly symmetric range that sacrifices one code point.
For 2-bit, sacrificing one of only 4 code points (25% capacity) for true symmetry ([-1, 1]) is expensive, so using the full range makes sense as a practical tradeoff. Just confirming this is intentional and that the QNN runtime handles this correctly with per_tensor_symmetric qscheme — since the scheme name doesn't quite match the range.
3. Missing ch_axis in get_16a2w_qnn_ptq_config weight spec
File:backends/qualcomm/quantizer/qconfig.py:241-246
The comparable get_16a4w_qnn_ptq_config (line 285-291) includes ch_axis=0 in its weight QuantizationSpec, but get_16a2w_qnn_ptq_config omits it. Since the comment states "2 bits weight quantization only supports per channel and symmetric," should ch_axis=0 be included here for consistency? This config is used as the default (non-per-channel) entry in QUANT_CONFIG_DICT, so it may be hit for ops that don't get the per-channel annotator.
4. CI failure: test_qnn_backend_16a2w_conv2d
The CI shows this test is failing. Worth investigating whether this is an issue with the conv2d lowering path for 2-bit, or an environment/setup issue on the CI runner.
5. Minor: test loop variable
File:backends/qualcomm/tests/test_qnn_delegate.py:2486
fori, per_channel, use_biasin [
(1, True, False),
(2, True, True),
]:
The loop starts the index at 1 instead of 0. This is cosmetic (only affects subTest output), but using enumerate like the conv2d test would be more consistent:
fori, (per_channel, use_bias) inenumerate([
(True, False),
(True, True),
]):
Summary
The implementation correctly extends the existing quantization infrastructure for 2-bit support. The key concern is item #1 (potential misclassification in the per-tensor path) — the rest are minor or confirmatory. The CI failure in #4 should be resolved before merging.
@linux-foundation-easycla
linux-foundation-easyclaBot
commented
May 30, 2026•

edited

Loading

Copy link


Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.
@jethroqti
jethroqti
commented
May 30, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@psiddh
psiddh
commented
Jun 2, 2026

Copy link


Copy Markdown

Contributor
@claude Re review the latest fixes and changes
@claude
claudeBot
commented
Jun 2, 2026•

edited

Loading

Copy link


Copy Markdown

Claude finished @psiddh's task in 1m 28s —— View job
Re-review: 2-bits quantization 16a2w support (after fixes)
  •  Read PR diff and latest changes
  •  Review backends/qualcomm/builders/node_visitor.py
  •  Review backends/qualcomm/quantizer/qconfig.py
  •  Review backends/qualcomm/quantizer/quantizer.py
  •  Review backends/qualcomm/quantizer/validators.py
  •  Review backends/qualcomm/tests/test_qnn_delegate.py
  •  Post review feedback
The fix commit (c3479b8) addresses the key issues raised in the first review. Here's the assessment of the current state:
Previously Raised Issues — Status
#IssueStatus
1Per-tensor misclassification of 2-bit as 4-bitFixed — assertion added in both node_visitor.py:278-282 and validators.py:286-290 to fail fast if 2-bit reaches per-tensor path
2Asymmetric range [-2, 1] with per_tensor_symmetricUnchanged — appears intentional for the hardware
3Missing ch_axis in get_16a2w_qnn_ptq_configNot addressed — acceptable since the assertion now guards this path
4CI test failureNeeds CI run to verify
5Test loop variableFixed — now uses enumerate() correctly
Current Code Assessment
The fixes look good. The assertion-based approach in the per-tensor paths is the right call — it's better to fail loudly than to silently misclassify.
One remaining minor concern in validators.py:306-309:
elifqspec.dtype==torch.int8andqspec.quant_max-qspec.quant_min<=3:
encoding_type= (
PyQnnManager.Qnn_QuantizationEncoding_t.QNN_QUANTIZATION_ENCODING_BW_AXIS_SCALE_OFFSET
)
elifqspec.dtype==torch.int8andqspec.quant_max-qspec.quant_min<=15:
This correctly handles 2-bit in the per-channel branch, but range_ was already computed earlier in the function (line 286). The per-channel branch recomputes qspec.quant_max - qspec.quant_min inline instead of using a local variable. This is purely cosmetic — not a bug.
Summary
The PR is in good shape after the fix commit. The core implementation correctly:
  • Extends per-channel config to handle torch.int2 with range [-2, 1]
  • Applies 2-bit masking (0x03) for tensor data encoding
  • Guards the per-tensor path with clear assertions
  • Properly registers use_16a2w in QUANT_CONFIG_DICT with the per-channel annotator
  • Tests both conv2d and linear with bias/no-bias variants
Remaining action item: CI approval needed to verify test_qnn_backend_16a2w_conv2d and test_qnn_backend_16a2w_linear pass on SM8850 hardware.
View job
@psiddh
psiddh
commented
Jun 4, 2026

Copy link


Copy Markdown

Contributor
This has merge conflicts, Pls rebase and this is good to go
@psiddhpsiddh
added
the 
module: qnn
Issues related to Qualcomm's QNN delegate and code under backends/qualcomm/
label
Jun 4, 2026
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
c3479b8 to
4fddea0CompareJune 5, 2026 01:12
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@psiddh
@psiddh
psiddh
commented
Jun 5, 2026

Copy link


Copy Markdown

Contributor
@jethroqti can you check if this failure is related to the PR ? pull / test-qnn-delegate-linux / linux-job (pull_request)
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
4fddea0 to
e96a504CompareJune 5, 2026 10:36
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@jethroqti
jethroqti
commented
Jun 6, 2026•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@psiddh

If I use QNN SDK version >=2.41, I did not find the "test_qnn_backend_16a2w_conv2d - AssertionError: False is not true : ref_output".

Could you try to use newer version QNN_SDK 2.41.0.251128?

Because I found HTP fixed an issue in the quantizer in v2.40.
shewu-quic
shewu-quic
reviewed
Jun 8, 2026
View reviewed changes
Comment threadbackends/qualcomm/tests/test_qnn_delegate.py
 shared_buffer=TestQNN.shared_buffer,
 )

 def test_qnn_backend_16a2w_conv2d(self):
@shewu-quicshewu-quicJun 8, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a skip condition based on qnn sdk version. Thanks
@jethroqti
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w
83e6bba
Summary:
1.Add 2-bits quantization basis 16a2w quantizer with standard symmetric
2.Support per channel and linear layers
3.Currently support soc model SM8850
Test plan:
python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_conv2d -b build-android -H ${HOST} -s ${SN} -m SM8850
python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_linear -b build-android -H ${HOST} -s ${SN} -m SM8850
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
e96a504 to
83e6bbaCompareJune 8, 2026 07:08
@jethroqti
jethroqti
commented
Jun 8, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@psiddh
psiddh
merged commit c4e3db0
into
pytorch:mainJun 8, 2026

176 of 178 checks passed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@psiddhpsiddhpsiddh approved these changes
@abhinaykukkadapuabhinaykukkadapuAwaiting requested review from abhinaykukkadapuabhinaykukkadapu is a code owner

+1 more reviewer

@shewu-quicshewu-quicshewu-quic left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.module: qnnIssues related to Qualcomm's QNN delegate and code under backends/qualcomm/release notes: qualcommChanges to the Qualcomm backend delegate

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jethroqti@psiddh@shewu-quic

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w - #19632
Merged
psiddh merged 1 commit into
pytorch:mainfrom 
CodeLinaro:dev1/quant/pcq2bit
Jun 8, 2026
Merged
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w#19632
psiddh merged 1 commit into
pytorch:mainfrom 
CodeLinaro:dev1/quant/pcq2bit
Conversation
@jethroqti
@jethroqtijethroqti
commented
May 18, 2026•

edited by pytorch-bot
Bot
Loading

Copy link


Copy Markdown

Contributor
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w
Summary:

1.Add 2-bits quantization basis 16a2w quantizer with standard symmetric

2.Support per channel and linear layers

3.Currently support soc model SM8850
Test plan:

python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_conv2d -b build-android -H ${HOST} -s ${SN} -m SM8850 python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_linear -b build-android -H ${HOST} -s ${SN} -m SM8850
cc @cccclai@cbilgin@abhinaykukkadapu
@pytorch-bot
pytorch-botBot
commented
May 18, 2026•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19632
Note: Links to docs will display an error until the docs builds have been completed.
❌ 1 New Failure, 1 Unrelated Failure
As of commit 83e6bba with merge base 02e57bd (image):
NEW FAILURE - The following job has failed:
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
May 18, 2026
@jethroqti
jethroqti
commented
May 18, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@pytorch-botpytorch-botBot
added
the 
release notes: qualcomm
Changes to the Qualcomm backend delegate
label
May 18, 2026
@jethroqti
jethroqti
commented
May 18, 2026

Copy link


Copy Markdown

ContributorAuthor
This PR is used to support 2-bits quantization basis 16a2w. Please take a look. Thanks.
@psiddh@haowhsu-quic@shewu-quic@winskuo-quic@DannyYuyang-quic
@psiddh
psiddh
commented
May 28, 2026

Copy link


Copy Markdown

Contributor
@claude pls review
@claude
claudeBot
commented
May 28, 2026•

edited

Loading

Copy link


Copy Markdown

Claude finished @psiddh's task in 2m 53s —— View job
Code Review: 2-bits quantization 16a2w support
  •  Read PR diff and changed files
  •  Review backends/qualcomm/builders/node_visitor.py
  •  Review backends/qualcomm/quantizer/qconfig.py
  •  Review backends/qualcomm/quantizer/quantizer.py
  •  Review backends/qualcomm/tests/test_qnn_delegate.py
  •  Post review feedback
Overall the changes are well-structured and follow the existing patterns for 4-bit quantization. A few items to address:
1. make_qnn_per_tensor_config may misclassify 2-bit as 4-bit
File:backends/qualcomm/builders/node_visitor.py:278-287
The per-channel path correctly checks for <= 3 before <= 15, but make_qnn_per_tensor_config still has only the 4-bit check:
# special case for 4 bitsif (
quant_config[QCOM_DTYPE] ==torch.int8andquant_config[QCOM_QUANT_MAX] -quant_config[QCOM_QUANT_MIN] <=15
):
quant_config[QCOM_BITWIDTH] =4
Since 1 - (-2) = 3 <= 15, a 2-bit tensor that reaches this per-tensor path would be incorrectly classified as 4-bit with QCOM_BITWIDTH = 4. I understand that 2-bit only supports per-channel currently, but adding a defensive 2-bit check here (matching the per-channel pattern) would prevent silent misclassification if the path is hit unexpectedly.
Fix this →
2. Weight quantization range [-2, 1] — intentional asymmetry?
File:backends/qualcomm/quantizer/qconfig.py:242-244
The 2-bit weight uses quant_min=-2, quant_max=1 with per_tensor_symmetric. This is the full 2-bit signed range (4 values: {-2, -1, 0, 1}), which is asymmetric around zero. By contrast, the 4-bit config uses [-7, 7] — a truly symmetric range that sacrifices one code point.
For 2-bit, sacrificing one of only 4 code points (25% capacity) for true symmetry ([-1, 1]) is expensive, so using the full range makes sense as a practical tradeoff. Just confirming this is intentional and that the QNN runtime handles this correctly with per_tensor_symmetric qscheme — since the scheme name doesn't quite match the range.
3. Missing ch_axis in get_16a2w_qnn_ptq_config weight spec
File:backends/qualcomm/quantizer/qconfig.py:241-246
The comparable get_16a4w_qnn_ptq_config (line 285-291) includes ch_axis=0 in its weight QuantizationSpec, but get_16a2w_qnn_ptq_config omits it. Since the comment states "2 bits weight quantization only supports per channel and symmetric," should ch_axis=0 be included here for consistency? This config is used as the default (non-per-channel) entry in QUANT_CONFIG_DICT, so it may be hit for ops that don't get the per-channel annotator.
4. CI failure: test_qnn_backend_16a2w_conv2d
The CI shows this test is failing. Worth investigating whether this is an issue with the conv2d lowering path for 2-bit, or an environment/setup issue on the CI runner.
5. Minor: test loop variable
File:backends/qualcomm/tests/test_qnn_delegate.py:2486
fori, per_channel, use_biasin [
(1, True, False),
(2, True, True),
]:
The loop starts the index at 1 instead of 0. This is cosmetic (only affects subTest output), but using enumerate like the conv2d test would be more consistent:
fori, (per_channel, use_bias) inenumerate([
(True, False),
(True, True),
]):
Summary
The implementation correctly extends the existing quantization infrastructure for 2-bit support. The key concern is item #1 (potential misclassification in the per-tensor path) — the rest are minor or confirmatory. The CI failure in #4 should be resolved before merging.
@linux-foundation-easycla
linux-foundation-easyclaBot
commented
May 30, 2026•

edited

Loading

Copy link


Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.
@jethroqti
jethroqti
commented
May 30, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@psiddh
psiddh
commented
Jun 2, 2026

Copy link


Copy Markdown

Contributor
@claude Re review the latest fixes and changes
@claude
claudeBot
commented
Jun 2, 2026•

edited

Loading

Copy link


Copy Markdown

Claude finished @psiddh's task in 1m 28s —— View job
Re-review: 2-bits quantization 16a2w support (after fixes)
  •  Read PR diff and latest changes
  •  Review backends/qualcomm/builders/node_visitor.py
  •  Review backends/qualcomm/quantizer/qconfig.py
  •  Review backends/qualcomm/quantizer/quantizer.py
  •  Review backends/qualcomm/quantizer/validators.py
  •  Review backends/qualcomm/tests/test_qnn_delegate.py
  •  Post review feedback
The fix commit (c3479b8) addresses the key issues raised in the first review. Here's the assessment of the current state:
Previously Raised Issues — Status
#IssueStatus
1Per-tensor misclassification of 2-bit as 4-bitFixed — assertion added in both node_visitor.py:278-282 and validators.py:286-290 to fail fast if 2-bit reaches per-tensor path
2Asymmetric range [-2, 1] with per_tensor_symmetricUnchanged — appears intentional for the hardware
3Missing ch_axis in get_16a2w_qnn_ptq_configNot addressed — acceptable since the assertion now guards this path
4CI test failureNeeds CI run to verify
5Test loop variableFixed — now uses enumerate() correctly
Current Code Assessment
The fixes look good. The assertion-based approach in the per-tensor paths is the right call — it's better to fail loudly than to silently misclassify.
One remaining minor concern in validators.py:306-309:
elifqspec.dtype==torch.int8andqspec.quant_max-qspec.quant_min<=3:
encoding_type= (
PyQnnManager.Qnn_QuantizationEncoding_t.QNN_QUANTIZATION_ENCODING_BW_AXIS_SCALE_OFFSET
)
elifqspec.dtype==torch.int8andqspec.quant_max-qspec.quant_min<=15:
This correctly handles 2-bit in the per-channel branch, but range_ was already computed earlier in the function (line 286). The per-channel branch recomputes qspec.quant_max - qspec.quant_min inline instead of using a local variable. This is purely cosmetic — not a bug.
Summary
The PR is in good shape after the fix commit. The core implementation correctly:
  • Extends per-channel config to handle torch.int2 with range [-2, 1]
  • Applies 2-bit masking (0x03) for tensor data encoding
  • Guards the per-tensor path with clear assertions
  • Properly registers use_16a2w in QUANT_CONFIG_DICT with the per-channel annotator
  • Tests both conv2d and linear with bias/no-bias variants
Remaining action item: CI approval needed to verify test_qnn_backend_16a2w_conv2d and test_qnn_backend_16a2w_linear pass on SM8850 hardware.
View job
@psiddh
psiddh
commented
Jun 4, 2026

Copy link


Copy Markdown

Contributor
This has merge conflicts, Pls rebase and this is good to go
@psiddhpsiddh
added
the 
module: qnn
Issues related to Qualcomm's QNN delegate and code under backends/qualcomm/
label
Jun 4, 2026
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
c3479b8 to
4fddea0CompareJune 5, 2026 01:12
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@psiddh
@psiddh
psiddh
commented
Jun 5, 2026

Copy link


Copy Markdown

Contributor
@jethroqti can you check if this failure is related to the PR ? pull / test-qnn-delegate-linux / linux-job (pull_request)
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
4fddea0 to
e96a504CompareJune 5, 2026 10:36
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@jethroqti
jethroqti
commented
Jun 6, 2026•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@psiddh

If I use QNN SDK version >=2.41, I did not find the "test_qnn_backend_16a2w_conv2d - AssertionError: False is not true : ref_output".

Could you try to use newer version QNN_SDK 2.41.0.251128?

Because I found HTP fixed an issue in the quantizer in v2.40.
shewu-quic
shewu-quic
reviewed
Jun 8, 2026
View reviewed changes
Comment threadbackends/qualcomm/tests/test_qnn_delegate.py
 shared_buffer=TestQNN.shared_buffer,
 )

 def test_qnn_backend_16a2w_conv2d(self):
@shewu-quicshewu-quicJun 8, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a skip condition based on qnn sdk version. Thanks
@jethroqti
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w
83e6bba
Summary:
1.Add 2-bits quantization basis 16a2w quantizer with standard symmetric
2.Support per channel and linear layers
3.Currently support soc model SM8850
Test plan:
python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_conv2d -b build-android -H ${HOST} -s ${SN} -m SM8850
python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_linear -b build-android -H ${HOST} -s ${SN} -m SM8850
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
e96a504 to
83e6bbaCompareJune 8, 2026 07:08
@jethroqti
jethroqti
commented
Jun 8, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@psiddh
psiddh
merged commit c4e3db0
into
pytorch:mainJun 8, 2026

176 of 178 checks passed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@psiddhpsiddhpsiddh approved these changes
@abhinaykukkadapuabhinaykukkadapuAwaiting requested review from abhinaykukkadapuabhinaykukkadapu is a code owner

+1 more reviewer

@shewu-quicshewu-quicshewu-quic left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.module: qnnIssues related to Qualcomm's QNN delegate and code under backends/qualcomm/release notes: qualcommChanges to the Qualcomm backend delegate

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jethroqti@psiddh@shewu-quic

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w - #19632
Merged
psiddh merged 1 commit into
pytorch:mainfrom 
CodeLinaro:dev1/quant/pcq2bit
Jun 8, 2026
Merged
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w#19632
psiddh merged 1 commit into
pytorch:mainfrom 
CodeLinaro:dev1/quant/pcq2bit
Conversation
@jethroqti
@jethroqtijethroqti
commented
May 18, 2026•

edited by pytorch-bot
Bot
Loading

Copy link


Copy Markdown

Contributor
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w
Summary:

1.Add 2-bits quantization basis 16a2w quantizer with standard symmetric

2.Support per channel and linear layers

3.Currently support soc model SM8850
Test plan:

python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_conv2d -b build-android -H ${HOST} -s ${SN} -m SM8850 python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_linear -b build-android -H ${HOST} -s ${SN} -m SM8850
cc @cccclai@cbilgin@abhinaykukkadapu
@pytorch-bot
pytorch-botBot
commented
May 18, 2026•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19632
Note: Links to docs will display an error until the docs builds have been completed.
❌ 1 New Failure, 1 Unrelated Failure
As of commit 83e6bba with merge base 02e57bd (image):
NEW FAILURE - The following job has failed:
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
May 18, 2026
@jethroqti
jethroqti
commented
May 18, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@pytorch-botpytorch-botBot
added
the 
release notes: qualcomm
Changes to the Qualcomm backend delegate
label
May 18, 2026
@jethroqti
jethroqti
commented
May 18, 2026

Copy link


Copy Markdown

ContributorAuthor
This PR is used to support 2-bits quantization basis 16a2w. Please take a look. Thanks.
@psiddh@haowhsu-quic@shewu-quic@winskuo-quic@DannyYuyang-quic
@psiddh
psiddh
commented
May 28, 2026

Copy link


Copy Markdown

Contributor
@claude pls review
@claude
claudeBot
commented
May 28, 2026•

edited

Loading

Copy link


Copy Markdown

Claude finished @psiddh's task in 2m 53s —— View job
Code Review: 2-bits quantization 16a2w support
  •  Read PR diff and changed files
  •  Review backends/qualcomm/builders/node_visitor.py
  •  Review backends/qualcomm/quantizer/qconfig.py
  •  Review backends/qualcomm/quantizer/quantizer.py
  •  Review backends/qualcomm/tests/test_qnn_delegate.py
  •  Post review feedback
Overall the changes are well-structured and follow the existing patterns for 4-bit quantization. A few items to address:
1. make_qnn_per_tensor_config may misclassify 2-bit as 4-bit
File:backends/qualcomm/builders/node_visitor.py:278-287
The per-channel path correctly checks for <= 3 before <= 15, but make_qnn_per_tensor_config still has only the 4-bit check:
# special case for 4 bitsif (
quant_config[QCOM_DTYPE] ==torch.int8andquant_config[QCOM_QUANT_MAX] -quant_config[QCOM_QUANT_MIN] <=15
):
quant_config[QCOM_BITWIDTH] =4
Since 1 - (-2) = 3 <= 15, a 2-bit tensor that reaches this per-tensor path would be incorrectly classified as 4-bit with QCOM_BITWIDTH = 4. I understand that 2-bit only supports per-channel currently, but adding a defensive 2-bit check here (matching the per-channel pattern) would prevent silent misclassification if the path is hit unexpectedly.
Fix this →
2. Weight quantization range [-2, 1] — intentional asymmetry?
File:backends/qualcomm/quantizer/qconfig.py:242-244
The 2-bit weight uses quant_min=-2, quant_max=1 with per_tensor_symmetric. This is the full 2-bit signed range (4 values: {-2, -1, 0, 1}), which is asymmetric around zero. By contrast, the 4-bit config uses [-7, 7] — a truly symmetric range that sacrifices one code point.
For 2-bit, sacrificing one of only 4 code points (25% capacity) for true symmetry ([-1, 1]) is expensive, so using the full range makes sense as a practical tradeoff. Just confirming this is intentional and that the QNN runtime handles this correctly with per_tensor_symmetric qscheme — since the scheme name doesn't quite match the range.
3. Missing ch_axis in get_16a2w_qnn_ptq_config weight spec
File:backends/qualcomm/quantizer/qconfig.py:241-246
The comparable get_16a4w_qnn_ptq_config (line 285-291) includes ch_axis=0 in its weight QuantizationSpec, but get_16a2w_qnn_ptq_config omits it. Since the comment states "2 bits weight quantization only supports per channel and symmetric," should ch_axis=0 be included here for consistency? This config is used as the default (non-per-channel) entry in QUANT_CONFIG_DICT, so it may be hit for ops that don't get the per-channel annotator.
4. CI failure: test_qnn_backend_16a2w_conv2d
The CI shows this test is failing. Worth investigating whether this is an issue with the conv2d lowering path for 2-bit, or an environment/setup issue on the CI runner.
5. Minor: test loop variable
File:backends/qualcomm/tests/test_qnn_delegate.py:2486
fori, per_channel, use_biasin [
(1, True, False),
(2, True, True),
]:
The loop starts the index at 1 instead of 0. This is cosmetic (only affects subTest output), but using enumerate like the conv2d test would be more consistent:
fori, (per_channel, use_bias) inenumerate([
(True, False),
(True, True),
]):
Summary
The implementation correctly extends the existing quantization infrastructure for 2-bit support. The key concern is item #1 (potential misclassification in the per-tensor path) — the rest are minor or confirmatory. The CI failure in #4 should be resolved before merging.
@linux-foundation-easycla
linux-foundation-easyclaBot
commented
May 30, 2026•

edited

Loading

Copy link


Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.
@jethroqti
jethroqti
commented
May 30, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@psiddh
psiddh
commented
Jun 2, 2026

Copy link


Copy Markdown

Contributor
@claude Re review the latest fixes and changes
@claude
claudeBot
commented
Jun 2, 2026•

edited

Loading

Copy link


Copy Markdown

Claude finished @psiddh's task in 1m 28s —— View job
Re-review: 2-bits quantization 16a2w support (after fixes)
  •  Read PR diff and latest changes
  •  Review backends/qualcomm/builders/node_visitor.py
  •  Review backends/qualcomm/quantizer/qconfig.py
  •  Review backends/qualcomm/quantizer/quantizer.py
  •  Review backends/qualcomm/quantizer/validators.py
  •  Review backends/qualcomm/tests/test_qnn_delegate.py
  •  Post review feedback
The fix commit (c3479b8) addresses the key issues raised in the first review. Here's the assessment of the current state:
Previously Raised Issues — Status
#IssueStatus
1Per-tensor misclassification of 2-bit as 4-bitFixed — assertion added in both node_visitor.py:278-282 and validators.py:286-290 to fail fast if 2-bit reaches per-tensor path
2Asymmetric range [-2, 1] with per_tensor_symmetricUnchanged — appears intentional for the hardware
3Missing ch_axis in get_16a2w_qnn_ptq_configNot addressed — acceptable since the assertion now guards this path
4CI test failureNeeds CI run to verify
5Test loop variableFixed — now uses enumerate() correctly
Current Code Assessment
The fixes look good. The assertion-based approach in the per-tensor paths is the right call — it's better to fail loudly than to silently misclassify.
One remaining minor concern in validators.py:306-309:
elifqspec.dtype==torch.int8andqspec.quant_max-qspec.quant_min<=3:
encoding_type= (
PyQnnManager.Qnn_QuantizationEncoding_t.QNN_QUANTIZATION_ENCODING_BW_AXIS_SCALE_OFFSET
)
elifqspec.dtype==torch.int8andqspec.quant_max-qspec.quant_min<=15:
This correctly handles 2-bit in the per-channel branch, but range_ was already computed earlier in the function (line 286). The per-channel branch recomputes qspec.quant_max - qspec.quant_min inline instead of using a local variable. This is purely cosmetic — not a bug.
Summary
The PR is in good shape after the fix commit. The core implementation correctly:
  • Extends per-channel config to handle torch.int2 with range [-2, 1]
  • Applies 2-bit masking (0x03) for tensor data encoding
  • Guards the per-tensor path with clear assertions
  • Properly registers use_16a2w in QUANT_CONFIG_DICT with the per-channel annotator
  • Tests both conv2d and linear with bias/no-bias variants
Remaining action item: CI approval needed to verify test_qnn_backend_16a2w_conv2d and test_qnn_backend_16a2w_linear pass on SM8850 hardware.
View job
@psiddh
psiddh
commented
Jun 4, 2026

Copy link


Copy Markdown

Contributor
This has merge conflicts, Pls rebase and this is good to go
@psiddhpsiddh
added
the 
module: qnn
Issues related to Qualcomm's QNN delegate and code under backends/qualcomm/
label
Jun 4, 2026
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
c3479b8 to
4fddea0CompareJune 5, 2026 01:12
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@psiddh
@psiddh
psiddh
commented
Jun 5, 2026

Copy link


Copy Markdown

Contributor
@jethroqti can you check if this failure is related to the PR ? pull / test-qnn-delegate-linux / linux-job (pull_request)
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
4fddea0 to
e96a504CompareJune 5, 2026 10:36
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@jethroqti
jethroqti
commented
Jun 6, 2026•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@psiddh

If I use QNN SDK version >=2.41, I did not find the "test_qnn_backend_16a2w_conv2d - AssertionError: False is not true : ref_output".

Could you try to use newer version QNN_SDK 2.41.0.251128?

Because I found HTP fixed an issue in the quantizer in v2.40.
shewu-quic
shewu-quic
reviewed
Jun 8, 2026
View reviewed changes
Comment threadbackends/qualcomm/tests/test_qnn_delegate.py
 shared_buffer=TestQNN.shared_buffer,
 )

 def test_qnn_backend_16a2w_conv2d(self):
@shewu-quicshewu-quicJun 8, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a skip condition based on qnn sdk version. Thanks
@jethroqti
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w
83e6bba
Summary:
1.Add 2-bits quantization basis 16a2w quantizer with standard symmetric
2.Support per channel and linear layers
3.Currently support soc model SM8850
Test plan:
python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_conv2d -b build-android -H ${HOST} -s ${SN} -m SM8850
python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_linear -b build-android -H ${HOST} -s ${SN} -m SM8850
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
e96a504 to
83e6bbaCompareJune 8, 2026 07:08
@jethroqti
jethroqti
commented
Jun 8, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@psiddh
psiddh
merged commit c4e3db0
into
pytorch:mainJun 8, 2026

176 of 178 checks passed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@psiddhpsiddhpsiddh approved these changes
@abhinaykukkadapuabhinaykukkadapuAwaiting requested review from abhinaykukkadapuabhinaykukkadapu is a code owner

+1 more reviewer

@shewu-quicshewu-quicshewu-quic left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.module: qnnIssues related to Qualcomm's QNN delegate and code under backends/qualcomm/release notes: qualcommChanges to the Qualcomm backend delegate

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jethroqti@psiddh@shewu-quic

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w - #19632
Merged
psiddh merged 1 commit into
pytorch:mainfrom 
CodeLinaro:dev1/quant/pcq2bit
Jun 8, 2026
Merged
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w#19632
psiddh merged 1 commit into
pytorch:mainfrom 
CodeLinaro:dev1/quant/pcq2bit
Conversation
@jethroqti
@jethroqtijethroqti
commented
May 18, 2026•

edited by pytorch-bot
Bot
Loading

Copy link


Copy Markdown

Contributor
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w
Summary:

1.Add 2-bits quantization basis 16a2w quantizer with standard symmetric

2.Support per channel and linear layers

3.Currently support soc model SM8850
Test plan:

python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_conv2d -b build-android -H ${HOST} -s ${SN} -m SM8850 python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_linear -b build-android -H ${HOST} -s ${SN} -m SM8850
cc @cccclai@cbilgin@abhinaykukkadapu
@pytorch-bot
pytorch-botBot
commented
May 18, 2026•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19632
Note: Links to docs will display an error until the docs builds have been completed.
❌ 1 New Failure, 1 Unrelated Failure
As of commit 83e6bba with merge base 02e57bd (image):
NEW FAILURE - The following job has failed:
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
May 18, 2026
@jethroqti
jethroqti
commented
May 18, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@pytorch-botpytorch-botBot
added
the 
release notes: qualcomm
Changes to the Qualcomm backend delegate
label
May 18, 2026
@jethroqti
jethroqti
commented
May 18, 2026

Copy link


Copy Markdown

ContributorAuthor
This PR is used to support 2-bits quantization basis 16a2w. Please take a look. Thanks.
@psiddh@haowhsu-quic@shewu-quic@winskuo-quic@DannyYuyang-quic
@psiddh
psiddh
commented
May 28, 2026

Copy link


Copy Markdown

Contributor
@claude pls review
@claude
claudeBot
commented
May 28, 2026•

edited

Loading

Copy link


Copy Markdown

Claude finished @psiddh's task in 2m 53s —— View job
Code Review: 2-bits quantization 16a2w support
  •  Read PR diff and changed files
  •  Review backends/qualcomm/builders/node_visitor.py
  •  Review backends/qualcomm/quantizer/qconfig.py
  •  Review backends/qualcomm/quantizer/quantizer.py
  •  Review backends/qualcomm/tests/test_qnn_delegate.py
  •  Post review feedback
Overall the changes are well-structured and follow the existing patterns for 4-bit quantization. A few items to address:
1. make_qnn_per_tensor_config may misclassify 2-bit as 4-bit
File:backends/qualcomm/builders/node_visitor.py:278-287
The per-channel path correctly checks for <= 3 before <= 15, but make_qnn_per_tensor_config still has only the 4-bit check:
# special case for 4 bitsif (
quant_config[QCOM_DTYPE] ==torch.int8andquant_config[QCOM_QUANT_MAX] -quant_config[QCOM_QUANT_MIN] <=15
):
quant_config[QCOM_BITWIDTH] =4
Since 1 - (-2) = 3 <= 15, a 2-bit tensor that reaches this per-tensor path would be incorrectly classified as 4-bit with QCOM_BITWIDTH = 4. I understand that 2-bit only supports per-channel currently, but adding a defensive 2-bit check here (matching the per-channel pattern) would prevent silent misclassification if the path is hit unexpectedly.
Fix this →
2. Weight quantization range [-2, 1] — intentional asymmetry?
File:backends/qualcomm/quantizer/qconfig.py:242-244
The 2-bit weight uses quant_min=-2, quant_max=1 with per_tensor_symmetric. This is the full 2-bit signed range (4 values: {-2, -1, 0, 1}), which is asymmetric around zero. By contrast, the 4-bit config uses [-7, 7] — a truly symmetric range that sacrifices one code point.
For 2-bit, sacrificing one of only 4 code points (25% capacity) for true symmetry ([-1, 1]) is expensive, so using the full range makes sense as a practical tradeoff. Just confirming this is intentional and that the QNN runtime handles this correctly with per_tensor_symmetric qscheme — since the scheme name doesn't quite match the range.
3. Missing ch_axis in get_16a2w_qnn_ptq_config weight spec
File:backends/qualcomm/quantizer/qconfig.py:241-246
The comparable get_16a4w_qnn_ptq_config (line 285-291) includes ch_axis=0 in its weight QuantizationSpec, but get_16a2w_qnn_ptq_config omits it. Since the comment states "2 bits weight quantization only supports per channel and symmetric," should ch_axis=0 be included here for consistency? This config is used as the default (non-per-channel) entry in QUANT_CONFIG_DICT, so it may be hit for ops that don't get the per-channel annotator.
4. CI failure: test_qnn_backend_16a2w_conv2d
The CI shows this test is failing. Worth investigating whether this is an issue with the conv2d lowering path for 2-bit, or an environment/setup issue on the CI runner.
5. Minor: test loop variable
File:backends/qualcomm/tests/test_qnn_delegate.py:2486
fori, per_channel, use_biasin [
(1, True, False),
(2, True, True),
]:
The loop starts the index at 1 instead of 0. This is cosmetic (only affects subTest output), but using enumerate like the conv2d test would be more consistent:
fori, (per_channel, use_bias) inenumerate([
(True, False),
(True, True),
]):
Summary
The implementation correctly extends the existing quantization infrastructure for 2-bit support. The key concern is item #1 (potential misclassification in the per-tensor path) — the rest are minor or confirmatory. The CI failure in #4 should be resolved before merging.
@linux-foundation-easycla
linux-foundation-easyclaBot
commented
May 30, 2026•

edited

Loading

Copy link


Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.
@jethroqti
jethroqti
commented
May 30, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@psiddh
psiddh
commented
Jun 2, 2026

Copy link


Copy Markdown

Contributor
@claude Re review the latest fixes and changes
@claude
claudeBot
commented
Jun 2, 2026•

edited

Loading

Copy link


Copy Markdown

Claude finished @psiddh's task in 1m 28s —— View job
Re-review: 2-bits quantization 16a2w support (after fixes)
  •  Read PR diff and latest changes
  •  Review backends/qualcomm/builders/node_visitor.py
  •  Review backends/qualcomm/quantizer/qconfig.py
  •  Review backends/qualcomm/quantizer/quantizer.py
  •  Review backends/qualcomm/quantizer/validators.py
  •  Review backends/qualcomm/tests/test_qnn_delegate.py
  •  Post review feedback
The fix commit (c3479b8) addresses the key issues raised in the first review. Here's the assessment of the current state:
Previously Raised Issues — Status
#IssueStatus
1Per-tensor misclassification of 2-bit as 4-bitFixed — assertion added in both node_visitor.py:278-282 and validators.py:286-290 to fail fast if 2-bit reaches per-tensor path
2Asymmetric range [-2, 1] with per_tensor_symmetricUnchanged — appears intentional for the hardware
3Missing ch_axis in get_16a2w_qnn_ptq_configNot addressed — acceptable since the assertion now guards this path
4CI test failureNeeds CI run to verify
5Test loop variableFixed — now uses enumerate() correctly
Current Code Assessment
The fixes look good. The assertion-based approach in the per-tensor paths is the right call — it's better to fail loudly than to silently misclassify.
One remaining minor concern in validators.py:306-309:
elifqspec.dtype==torch.int8andqspec.quant_max-qspec.quant_min<=3:
encoding_type= (
PyQnnManager.Qnn_QuantizationEncoding_t.QNN_QUANTIZATION_ENCODING_BW_AXIS_SCALE_OFFSET
)
elifqspec.dtype==torch.int8andqspec.quant_max-qspec.quant_min<=15:
This correctly handles 2-bit in the per-channel branch, but range_ was already computed earlier in the function (line 286). The per-channel branch recomputes qspec.quant_max - qspec.quant_min inline instead of using a local variable. This is purely cosmetic — not a bug.
Summary
The PR is in good shape after the fix commit. The core implementation correctly:
  • Extends per-channel config to handle torch.int2 with range [-2, 1]
  • Applies 2-bit masking (0x03) for tensor data encoding
  • Guards the per-tensor path with clear assertions
  • Properly registers use_16a2w in QUANT_CONFIG_DICT with the per-channel annotator
  • Tests both conv2d and linear with bias/no-bias variants
Remaining action item: CI approval needed to verify test_qnn_backend_16a2w_conv2d and test_qnn_backend_16a2w_linear pass on SM8850 hardware.
View job
@psiddh
psiddh
commented
Jun 4, 2026

Copy link


Copy Markdown

Contributor
This has merge conflicts, Pls rebase and this is good to go
@psiddhpsiddh
added
the 
module: qnn
Issues related to Qualcomm's QNN delegate and code under backends/qualcomm/
label
Jun 4, 2026
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
c3479b8 to
4fddea0CompareJune 5, 2026 01:12
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@psiddh
@psiddh
psiddh
commented
Jun 5, 2026

Copy link


Copy Markdown

Contributor
@jethroqti can you check if this failure is related to the PR ? pull / test-qnn-delegate-linux / linux-job (pull_request)
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
4fddea0 to
e96a504CompareJune 5, 2026 10:36
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@jethroqti
jethroqti
commented
Jun 6, 2026•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@psiddh

If I use QNN SDK version >=2.41, I did not find the "test_qnn_backend_16a2w_conv2d - AssertionError: False is not true : ref_output".

Could you try to use newer version QNN_SDK 2.41.0.251128?

Because I found HTP fixed an issue in the quantizer in v2.40.
shewu-quic
shewu-quic
reviewed
Jun 8, 2026
View reviewed changes
Comment threadbackends/qualcomm/tests/test_qnn_delegate.py
 shared_buffer=TestQNN.shared_buffer,
 )

 def test_qnn_backend_16a2w_conv2d(self):
@shewu-quicshewu-quicJun 8, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a skip condition based on qnn sdk version. Thanks
@jethroqti
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w
83e6bba
Summary:
1.Add 2-bits quantization basis 16a2w quantizer with standard symmetric
2.Support per channel and linear layers
3.Currently support soc model SM8850
Test plan:
python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_conv2d -b build-android -H ${HOST} -s ${SN} -m SM8850
python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_linear -b build-android -H ${HOST} -s ${SN} -m SM8850
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
e96a504 to
83e6bbaCompareJune 8, 2026 07:08
@jethroqti
jethroqti
commented
Jun 8, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@psiddh
psiddh
merged commit c4e3db0
into
pytorch:mainJun 8, 2026

176 of 178 checks passed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@psiddhpsiddhpsiddh approved these changes
@abhinaykukkadapuabhinaykukkadapuAwaiting requested review from abhinaykukkadapuabhinaykukkadapu is a code owner

+1 more reviewer

@shewu-quicshewu-quicshewu-quic left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.module: qnnIssues related to Qualcomm's QNN delegate and code under backends/qualcomm/release notes: qualcommChanges to the Qualcomm backend delegate

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jethroqti@psiddh@shewu-quic

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w - #19632
Merged
psiddh merged 1 commit into
pytorch:mainfrom 
CodeLinaro:dev1/quant/pcq2bit
Jun 8, 2026
Merged
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w#19632
psiddh merged 1 commit into
pytorch:mainfrom 
CodeLinaro:dev1/quant/pcq2bit
Conversation
@jethroqti
@jethroqtijethroqti
commented
May 18, 2026•

edited by pytorch-bot
Bot
Loading

Copy link


Copy Markdown

Contributor
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w
Summary:

1.Add 2-bits quantization basis 16a2w quantizer with standard symmetric

2.Support per channel and linear layers

3.Currently support soc model SM8850
Test plan:

python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_conv2d -b build-android -H ${HOST} -s ${SN} -m SM8850 python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_linear -b build-android -H ${HOST} -s ${SN} -m SM8850
cc @cccclai@cbilgin@abhinaykukkadapu
@pytorch-bot
pytorch-botBot
commented
May 18, 2026•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19632
Note: Links to docs will display an error until the docs builds have been completed.
❌ 1 New Failure, 1 Unrelated Failure
As of commit 83e6bba with merge base 02e57bd (image):
NEW FAILURE - The following job has failed:
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
May 18, 2026
@jethroqti
jethroqti
commented
May 18, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@pytorch-botpytorch-botBot
added
the 
release notes: qualcomm
Changes to the Qualcomm backend delegate
label
May 18, 2026
@jethroqti
jethroqti
commented
May 18, 2026

Copy link


Copy Markdown

ContributorAuthor
This PR is used to support 2-bits quantization basis 16a2w. Please take a look. Thanks.
@psiddh@haowhsu-quic@shewu-quic@winskuo-quic@DannyYuyang-quic
@psiddh
psiddh
commented
May 28, 2026

Copy link


Copy Markdown

Contributor
@claude pls review
@claude
claudeBot
commented
May 28, 2026•

edited

Loading

Copy link


Copy Markdown

Claude finished @psiddh's task in 2m 53s —— View job
Code Review: 2-bits quantization 16a2w support
  •  Read PR diff and changed files
  •  Review backends/qualcomm/builders/node_visitor.py
  •  Review backends/qualcomm/quantizer/qconfig.py
  •  Review backends/qualcomm/quantizer/quantizer.py
  •  Review backends/qualcomm/tests/test_qnn_delegate.py
  •  Post review feedback
Overall the changes are well-structured and follow the existing patterns for 4-bit quantization. A few items to address:
1. make_qnn_per_tensor_config may misclassify 2-bit as 4-bit
File:backends/qualcomm/builders/node_visitor.py:278-287
The per-channel path correctly checks for <= 3 before <= 15, but make_qnn_per_tensor_config still has only the 4-bit check:
# special case for 4 bitsif (
quant_config[QCOM_DTYPE] ==torch.int8andquant_config[QCOM_QUANT_MAX] -quant_config[QCOM_QUANT_MIN] <=15
):
quant_config[QCOM_BITWIDTH] =4
Since 1 - (-2) = 3 <= 15, a 2-bit tensor that reaches this per-tensor path would be incorrectly classified as 4-bit with QCOM_BITWIDTH = 4. I understand that 2-bit only supports per-channel currently, but adding a defensive 2-bit check here (matching the per-channel pattern) would prevent silent misclassification if the path is hit unexpectedly.
Fix this →
2. Weight quantization range [-2, 1] — intentional asymmetry?
File:backends/qualcomm/quantizer/qconfig.py:242-244
The 2-bit weight uses quant_min=-2, quant_max=1 with per_tensor_symmetric. This is the full 2-bit signed range (4 values: {-2, -1, 0, 1}), which is asymmetric around zero. By contrast, the 4-bit config uses [-7, 7] — a truly symmetric range that sacrifices one code point.
For 2-bit, sacrificing one of only 4 code points (25% capacity) for true symmetry ([-1, 1]) is expensive, so using the full range makes sense as a practical tradeoff. Just confirming this is intentional and that the QNN runtime handles this correctly with per_tensor_symmetric qscheme — since the scheme name doesn't quite match the range.
3. Missing ch_axis in get_16a2w_qnn_ptq_config weight spec
File:backends/qualcomm/quantizer/qconfig.py:241-246
The comparable get_16a4w_qnn_ptq_config (line 285-291) includes ch_axis=0 in its weight QuantizationSpec, but get_16a2w_qnn_ptq_config omits it. Since the comment states "2 bits weight quantization only supports per channel and symmetric," should ch_axis=0 be included here for consistency? This config is used as the default (non-per-channel) entry in QUANT_CONFIG_DICT, so it may be hit for ops that don't get the per-channel annotator.
4. CI failure: test_qnn_backend_16a2w_conv2d
The CI shows this test is failing. Worth investigating whether this is an issue with the conv2d lowering path for 2-bit, or an environment/setup issue on the CI runner.
5. Minor: test loop variable
File:backends/qualcomm/tests/test_qnn_delegate.py:2486
fori, per_channel, use_biasin [
(1, True, False),
(2, True, True),
]:
The loop starts the index at 1 instead of 0. This is cosmetic (only affects subTest output), but using enumerate like the conv2d test would be more consistent:
fori, (per_channel, use_bias) inenumerate([
(True, False),
(True, True),
]):
Summary
The implementation correctly extends the existing quantization infrastructure for 2-bit support. The key concern is item #1 (potential misclassification in the per-tensor path) — the rest are minor or confirmatory. The CI failure in #4 should be resolved before merging.
@linux-foundation-easycla
linux-foundation-easyclaBot
commented
May 30, 2026•

edited

Loading

Copy link


Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.
@jethroqti
jethroqti
commented
May 30, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@psiddh
psiddh
commented
Jun 2, 2026

Copy link


Copy Markdown

Contributor
@claude Re review the latest fixes and changes
@claude
claudeBot
commented
Jun 2, 2026•

edited

Loading

Copy link


Copy Markdown

Claude finished @psiddh's task in 1m 28s —— View job
Re-review: 2-bits quantization 16a2w support (after fixes)
  •  Read PR diff and latest changes
  •  Review backends/qualcomm/builders/node_visitor.py
  •  Review backends/qualcomm/quantizer/qconfig.py
  •  Review backends/qualcomm/quantizer/quantizer.py
  •  Review backends/qualcomm/quantizer/validators.py
  •  Review backends/qualcomm/tests/test_qnn_delegate.py
  •  Post review feedback
The fix commit (c3479b8) addresses the key issues raised in the first review. Here's the assessment of the current state:
Previously Raised Issues — Status
#IssueStatus
1Per-tensor misclassification of 2-bit as 4-bitFixed — assertion added in both node_visitor.py:278-282 and validators.py:286-290 to fail fast if 2-bit reaches per-tensor path
2Asymmetric range [-2, 1] with per_tensor_symmetricUnchanged — appears intentional for the hardware
3Missing ch_axis in get_16a2w_qnn_ptq_configNot addressed — acceptable since the assertion now guards this path
4CI test failureNeeds CI run to verify
5Test loop variableFixed — now uses enumerate() correctly
Current Code Assessment
The fixes look good. The assertion-based approach in the per-tensor paths is the right call — it's better to fail loudly than to silently misclassify.
One remaining minor concern in validators.py:306-309:
elifqspec.dtype==torch.int8andqspec.quant_max-qspec.quant_min<=3:
encoding_type= (
PyQnnManager.Qnn_QuantizationEncoding_t.QNN_QUANTIZATION_ENCODING_BW_AXIS_SCALE_OFFSET
)
elifqspec.dtype==torch.int8andqspec.quant_max-qspec.quant_min<=15:
This correctly handles 2-bit in the per-channel branch, but range_ was already computed earlier in the function (line 286). The per-channel branch recomputes qspec.quant_max - qspec.quant_min inline instead of using a local variable. This is purely cosmetic — not a bug.
Summary
The PR is in good shape after the fix commit. The core implementation correctly:
  • Extends per-channel config to handle torch.int2 with range [-2, 1]
  • Applies 2-bit masking (0x03) for tensor data encoding
  • Guards the per-tensor path with clear assertions
  • Properly registers use_16a2w in QUANT_CONFIG_DICT with the per-channel annotator
  • Tests both conv2d and linear with bias/no-bias variants
Remaining action item: CI approval needed to verify test_qnn_backend_16a2w_conv2d and test_qnn_backend_16a2w_linear pass on SM8850 hardware.
View job
@psiddh
psiddh
commented
Jun 4, 2026

Copy link


Copy Markdown

Contributor
This has merge conflicts, Pls rebase and this is good to go
@psiddhpsiddh
added
the 
module: qnn
Issues related to Qualcomm's QNN delegate and code under backends/qualcomm/
label
Jun 4, 2026
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
c3479b8 to
4fddea0CompareJune 5, 2026 01:12
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@psiddh
@psiddh
psiddh
commented
Jun 5, 2026

Copy link


Copy Markdown

Contributor
@jethroqti can you check if this failure is related to the PR ? pull / test-qnn-delegate-linux / linux-job (pull_request)
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
4fddea0 to
e96a504CompareJune 5, 2026 10:36
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@jethroqti
jethroqti
commented
Jun 6, 2026•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@psiddh

If I use QNN SDK version >=2.41, I did not find the "test_qnn_backend_16a2w_conv2d - AssertionError: False is not true : ref_output".

Could you try to use newer version QNN_SDK 2.41.0.251128?

Because I found HTP fixed an issue in the quantizer in v2.40.
shewu-quic
shewu-quic
reviewed
Jun 8, 2026
View reviewed changes
Comment threadbackends/qualcomm/tests/test_qnn_delegate.py
 shared_buffer=TestQNN.shared_buffer,
 )

 def test_qnn_backend_16a2w_conv2d(self):
@shewu-quicshewu-quicJun 8, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a skip condition based on qnn sdk version. Thanks
@jethroqti
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w
83e6bba
Summary:
1.Add 2-bits quantization basis 16a2w quantizer with standard symmetric
2.Support per channel and linear layers
3.Currently support soc model SM8850
Test plan:
python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_conv2d -b build-android -H ${HOST} -s ${SN} -m SM8850
python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_linear -b build-android -H ${HOST} -s ${SN} -m SM8850
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
e96a504 to
83e6bbaCompareJune 8, 2026 07:08
@jethroqti
jethroqti
commented
Jun 8, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@psiddh
psiddh
merged commit c4e3db0
into
pytorch:mainJun 8, 2026

176 of 178 checks passed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@psiddhpsiddhpsiddh approved these changes
@abhinaykukkadapuabhinaykukkadapuAwaiting requested review from abhinaykukkadapuabhinaykukkadapu is a code owner

+1 more reviewer

@shewu-quicshewu-quicshewu-quic left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.module: qnnIssues related to Qualcomm's QNN delegate and code under backends/qualcomm/release notes: qualcommChanges to the Qualcomm backend delegate

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jethroqti@psiddh@shewu-quic

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w - #19632
Merged
psiddh merged 1 commit into
pytorch:mainfrom 
CodeLinaro:dev1/quant/pcq2bit
Jun 8, 2026
Merged
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w#19632
psiddh merged 1 commit into
pytorch:mainfrom 
CodeLinaro:dev1/quant/pcq2bit
Conversation
@jethroqti
@jethroqtijethroqti
commented
May 18, 2026•

edited by pytorch-bot
Bot
Loading

Copy link


Copy Markdown

Contributor
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w
Summary:

1.Add 2-bits quantization basis 16a2w quantizer with standard symmetric

2.Support per channel and linear layers

3.Currently support soc model SM8850
Test plan:

python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_conv2d -b build-android -H ${HOST} -s ${SN} -m SM8850 python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_linear -b build-android -H ${HOST} -s ${SN} -m SM8850
cc @cccclai@cbilgin@abhinaykukkadapu
@pytorch-bot
pytorch-botBot
commented
May 18, 2026•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19632
Note: Links to docs will display an error until the docs builds have been completed.
❌ 1 New Failure, 1 Unrelated Failure
As of commit 83e6bba with merge base 02e57bd (image):
NEW FAILURE - The following job has failed:
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
May 18, 2026
@jethroqti
jethroqti
commented
May 18, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@pytorch-botpytorch-botBot
added
the 
release notes: qualcomm
Changes to the Qualcomm backend delegate
label
May 18, 2026
@jethroqti
jethroqti
commented
May 18, 2026

Copy link


Copy Markdown

ContributorAuthor
This PR is used to support 2-bits quantization basis 16a2w. Please take a look. Thanks.
@psiddh@haowhsu-quic@shewu-quic@winskuo-quic@DannyYuyang-quic
@psiddh
psiddh
commented
May 28, 2026

Copy link


Copy Markdown

Contributor
@claude pls review
@claude
claudeBot
commented
May 28, 2026•

edited

Loading

Copy link


Copy Markdown

Claude finished @psiddh's task in 2m 53s —— View job
Code Review: 2-bits quantization 16a2w support
  •  Read PR diff and changed files
  •  Review backends/qualcomm/builders/node_visitor.py
  •  Review backends/qualcomm/quantizer/qconfig.py
  •  Review backends/qualcomm/quantizer/quantizer.py
  •  Review backends/qualcomm/tests/test_qnn_delegate.py
  •  Post review feedback
Overall the changes are well-structured and follow the existing patterns for 4-bit quantization. A few items to address:
1. make_qnn_per_tensor_config may misclassify 2-bit as 4-bit
File:backends/qualcomm/builders/node_visitor.py:278-287
The per-channel path correctly checks for <= 3 before <= 15, but make_qnn_per_tensor_config still has only the 4-bit check:
# special case for 4 bitsif (
quant_config[QCOM_DTYPE] ==torch.int8andquant_config[QCOM_QUANT_MAX] -quant_config[QCOM_QUANT_MIN] <=15
):
quant_config[QCOM_BITWIDTH] =4
Since 1 - (-2) = 3 <= 15, a 2-bit tensor that reaches this per-tensor path would be incorrectly classified as 4-bit with QCOM_BITWIDTH = 4. I understand that 2-bit only supports per-channel currently, but adding a defensive 2-bit check here (matching the per-channel pattern) would prevent silent misclassification if the path is hit unexpectedly.
Fix this →
2. Weight quantization range [-2, 1] — intentional asymmetry?
File:backends/qualcomm/quantizer/qconfig.py:242-244
The 2-bit weight uses quant_min=-2, quant_max=1 with per_tensor_symmetric. This is the full 2-bit signed range (4 values: {-2, -1, 0, 1}), which is asymmetric around zero. By contrast, the 4-bit config uses [-7, 7] — a truly symmetric range that sacrifices one code point.
For 2-bit, sacrificing one of only 4 code points (25% capacity) for true symmetry ([-1, 1]) is expensive, so using the full range makes sense as a practical tradeoff. Just confirming this is intentional and that the QNN runtime handles this correctly with per_tensor_symmetric qscheme — since the scheme name doesn't quite match the range.
3. Missing ch_axis in get_16a2w_qnn_ptq_config weight spec
File:backends/qualcomm/quantizer/qconfig.py:241-246
The comparable get_16a4w_qnn_ptq_config (line 285-291) includes ch_axis=0 in its weight QuantizationSpec, but get_16a2w_qnn_ptq_config omits it. Since the comment states "2 bits weight quantization only supports per channel and symmetric," should ch_axis=0 be included here for consistency? This config is used as the default (non-per-channel) entry in QUANT_CONFIG_DICT, so it may be hit for ops that don't get the per-channel annotator.
4. CI failure: test_qnn_backend_16a2w_conv2d
The CI shows this test is failing. Worth investigating whether this is an issue with the conv2d lowering path for 2-bit, or an environment/setup issue on the CI runner.
5. Minor: test loop variable
File:backends/qualcomm/tests/test_qnn_delegate.py:2486
fori, per_channel, use_biasin [
(1, True, False),
(2, True, True),
]:
The loop starts the index at 1 instead of 0. This is cosmetic (only affects subTest output), but using enumerate like the conv2d test would be more consistent:
fori, (per_channel, use_bias) inenumerate([
(True, False),
(True, True),
]):
Summary
The implementation correctly extends the existing quantization infrastructure for 2-bit support. The key concern is item #1 (potential misclassification in the per-tensor path) — the rest are minor or confirmatory. The CI failure in #4 should be resolved before merging.
@linux-foundation-easycla
linux-foundation-easyclaBot
commented
May 30, 2026•

edited

Loading

Copy link


Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.
@jethroqti
jethroqti
commented
May 30, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@psiddh
psiddh
commented
Jun 2, 2026

Copy link


Copy Markdown

Contributor
@claude Re review the latest fixes and changes
@claude
claudeBot
commented
Jun 2, 2026•

edited

Loading

Copy link


Copy Markdown

Claude finished @psiddh's task in 1m 28s —— View job
Re-review: 2-bits quantization 16a2w support (after fixes)
  •  Read PR diff and latest changes
  •  Review backends/qualcomm/builders/node_visitor.py
  •  Review backends/qualcomm/quantizer/qconfig.py
  •  Review backends/qualcomm/quantizer/quantizer.py
  •  Review backends/qualcomm/quantizer/validators.py
  •  Review backends/qualcomm/tests/test_qnn_delegate.py
  •  Post review feedback
The fix commit (c3479b8) addresses the key issues raised in the first review. Here's the assessment of the current state:
Previously Raised Issues — Status
#IssueStatus
1Per-tensor misclassification of 2-bit as 4-bitFixed — assertion added in both node_visitor.py:278-282 and validators.py:286-290 to fail fast if 2-bit reaches per-tensor path
2Asymmetric range [-2, 1] with per_tensor_symmetricUnchanged — appears intentional for the hardware
3Missing ch_axis in get_16a2w_qnn_ptq_configNot addressed — acceptable since the assertion now guards this path
4CI test failureNeeds CI run to verify
5Test loop variableFixed — now uses enumerate() correctly
Current Code Assessment
The fixes look good. The assertion-based approach in the per-tensor paths is the right call — it's better to fail loudly than to silently misclassify.
One remaining minor concern in validators.py:306-309:
elifqspec.dtype==torch.int8andqspec.quant_max-qspec.quant_min<=3:
encoding_type= (
PyQnnManager.Qnn_QuantizationEncoding_t.QNN_QUANTIZATION_ENCODING_BW_AXIS_SCALE_OFFSET
)
elifqspec.dtype==torch.int8andqspec.quant_max-qspec.quant_min<=15:
This correctly handles 2-bit in the per-channel branch, but range_ was already computed earlier in the function (line 286). The per-channel branch recomputes qspec.quant_max - qspec.quant_min inline instead of using a local variable. This is purely cosmetic — not a bug.
Summary
The PR is in good shape after the fix commit. The core implementation correctly:
  • Extends per-channel config to handle torch.int2 with range [-2, 1]
  • Applies 2-bit masking (0x03) for tensor data encoding
  • Guards the per-tensor path with clear assertions
  • Properly registers use_16a2w in QUANT_CONFIG_DICT with the per-channel annotator
  • Tests both conv2d and linear with bias/no-bias variants
Remaining action item: CI approval needed to verify test_qnn_backend_16a2w_conv2d and test_qnn_backend_16a2w_linear pass on SM8850 hardware.
View job
@psiddh
psiddh
commented
Jun 4, 2026

Copy link


Copy Markdown

Contributor
This has merge conflicts, Pls rebase and this is good to go
@psiddhpsiddh
added
the 
module: qnn
Issues related to Qualcomm's QNN delegate and code under backends/qualcomm/
label
Jun 4, 2026
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
c3479b8 to
4fddea0CompareJune 5, 2026 01:12
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@psiddh
@psiddh
psiddh
commented
Jun 5, 2026

Copy link


Copy Markdown

Contributor
@jethroqti can you check if this failure is related to the PR ? pull / test-qnn-delegate-linux / linux-job (pull_request)
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
4fddea0 to
e96a504CompareJune 5, 2026 10:36
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@jethroqti
jethroqti
commented
Jun 6, 2026•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@psiddh

If I use QNN SDK version >=2.41, I did not find the "test_qnn_backend_16a2w_conv2d - AssertionError: False is not true : ref_output".

Could you try to use newer version QNN_SDK 2.41.0.251128?

Because I found HTP fixed an issue in the quantizer in v2.40.
shewu-quic
shewu-quic
reviewed
Jun 8, 2026
View reviewed changes
Comment threadbackends/qualcomm/tests/test_qnn_delegate.py
 shared_buffer=TestQNN.shared_buffer,
 )

 def test_qnn_backend_16a2w_conv2d(self):
@shewu-quicshewu-quicJun 8, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a skip condition based on qnn sdk version. Thanks
@jethroqti
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w
83e6bba
Summary:
1.Add 2-bits quantization basis 16a2w quantizer with standard symmetric
2.Support per channel and linear layers
3.Currently support soc model SM8850
Test plan:
python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_conv2d -b build-android -H ${HOST} -s ${SN} -m SM8850
python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_linear -b build-android -H ${HOST} -s ${SN} -m SM8850
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
e96a504 to
83e6bbaCompareJune 8, 2026 07:08
@jethroqti
jethroqti
commented
Jun 8, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@psiddh
psiddh
merged commit c4e3db0
into
pytorch:mainJun 8, 2026

176 of 178 checks passed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@psiddhpsiddhpsiddh approved these changes
@abhinaykukkadapuabhinaykukkadapuAwaiting requested review from abhinaykukkadapuabhinaykukkadapu is a code owner

+1 more reviewer

@shewu-quicshewu-quicshewu-quic left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.module: qnnIssues related to Qualcomm's QNN delegate and code under backends/qualcomm/release notes: qualcommChanges to the Qualcomm backend delegate

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jethroqti@psiddh@shewu-quic

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w - #19632
Merged
psiddh merged 1 commit into
pytorch:mainfrom 
CodeLinaro:dev1/quant/pcq2bit
Jun 8, 2026
Merged
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w#19632
psiddh merged 1 commit into
pytorch:mainfrom 
CodeLinaro:dev1/quant/pcq2bit
Conversation
@jethroqti
@jethroqtijethroqti
commented
May 18, 2026•

edited by pytorch-bot
Bot
Loading

Copy link


Copy Markdown

Contributor
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w
Summary:

1.Add 2-bits quantization basis 16a2w quantizer with standard symmetric

2.Support per channel and linear layers

3.Currently support soc model SM8850
Test plan:

python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_conv2d -b build-android -H ${HOST} -s ${SN} -m SM8850 python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_linear -b build-android -H ${HOST} -s ${SN} -m SM8850
cc @cccclai@cbilgin@abhinaykukkadapu
@pytorch-bot
pytorch-botBot
commented
May 18, 2026•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/19632
Note: Links to docs will display an error until the docs builds have been completed.
❌ 1 New Failure, 1 Unrelated Failure
As of commit 83e6bba with merge base 02e57bd (image):
NEW FAILURE - The following job has failed:
BROKEN TRUNK - The following job failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
@meta-clameta-claBot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
May 18, 2026
@jethroqti
jethroqti
commented
May 18, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@pytorch-botpytorch-botBot
added
the 
release notes: qualcomm
Changes to the Qualcomm backend delegate
label
May 18, 2026
@jethroqti
jethroqti
commented
May 18, 2026

Copy link


Copy Markdown

ContributorAuthor
This PR is used to support 2-bits quantization basis 16a2w. Please take a look. Thanks.
@psiddh@haowhsu-quic@shewu-quic@winskuo-quic@DannyYuyang-quic
@psiddh
psiddh
commented
May 28, 2026

Copy link


Copy Markdown

Contributor
@claude pls review
@claude
claudeBot
commented
May 28, 2026•

edited

Loading

Copy link


Copy Markdown

Claude finished @psiddh's task in 2m 53s —— View job
Code Review: 2-bits quantization 16a2w support
  •  Read PR diff and changed files
  •  Review backends/qualcomm/builders/node_visitor.py
  •  Review backends/qualcomm/quantizer/qconfig.py
  •  Review backends/qualcomm/quantizer/quantizer.py
  •  Review backends/qualcomm/tests/test_qnn_delegate.py
  •  Post review feedback
Overall the changes are well-structured and follow the existing patterns for 4-bit quantization. A few items to address:
1. make_qnn_per_tensor_config may misclassify 2-bit as 4-bit
File:backends/qualcomm/builders/node_visitor.py:278-287
The per-channel path correctly checks for <= 3 before <= 15, but make_qnn_per_tensor_config still has only the 4-bit check:
# special case for 4 bitsif (
quant_config[QCOM_DTYPE] ==torch.int8andquant_config[QCOM_QUANT_MAX] -quant_config[QCOM_QUANT_MIN] <=15
):
quant_config[QCOM_BITWIDTH] =4
Since 1 - (-2) = 3 <= 15, a 2-bit tensor that reaches this per-tensor path would be incorrectly classified as 4-bit with QCOM_BITWIDTH = 4. I understand that 2-bit only supports per-channel currently, but adding a defensive 2-bit check here (matching the per-channel pattern) would prevent silent misclassification if the path is hit unexpectedly.
Fix this →
2. Weight quantization range [-2, 1] — intentional asymmetry?
File:backends/qualcomm/quantizer/qconfig.py:242-244
The 2-bit weight uses quant_min=-2, quant_max=1 with per_tensor_symmetric. This is the full 2-bit signed range (4 values: {-2, -1, 0, 1}), which is asymmetric around zero. By contrast, the 4-bit config uses [-7, 7] — a truly symmetric range that sacrifices one code point.
For 2-bit, sacrificing one of only 4 code points (25% capacity) for true symmetry ([-1, 1]) is expensive, so using the full range makes sense as a practical tradeoff. Just confirming this is intentional and that the QNN runtime handles this correctly with per_tensor_symmetric qscheme — since the scheme name doesn't quite match the range.
3. Missing ch_axis in get_16a2w_qnn_ptq_config weight spec
File:backends/qualcomm/quantizer/qconfig.py:241-246
The comparable get_16a4w_qnn_ptq_config (line 285-291) includes ch_axis=0 in its weight QuantizationSpec, but get_16a2w_qnn_ptq_config omits it. Since the comment states "2 bits weight quantization only supports per channel and symmetric," should ch_axis=0 be included here for consistency? This config is used as the default (non-per-channel) entry in QUANT_CONFIG_DICT, so it may be hit for ops that don't get the per-channel annotator.
4. CI failure: test_qnn_backend_16a2w_conv2d
The CI shows this test is failing. Worth investigating whether this is an issue with the conv2d lowering path for 2-bit, or an environment/setup issue on the CI runner.
5. Minor: test loop variable
File:backends/qualcomm/tests/test_qnn_delegate.py:2486
fori, per_channel, use_biasin [
(1, True, False),
(2, True, True),
]:
The loop starts the index at 1 instead of 0. This is cosmetic (only affects subTest output), but using enumerate like the conv2d test would be more consistent:
fori, (per_channel, use_bias) inenumerate([
(True, False),
(True, True),
]):
Summary
The implementation correctly extends the existing quantization infrastructure for 2-bit support. The key concern is item #1 (potential misclassification in the per-tensor path) — the rest are minor or confirmatory. The CI failure in #4 should be resolved before merging.
@linux-foundation-easycla
linux-foundation-easyclaBot
commented
May 30, 2026•

edited

Loading

Copy link


Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.
@jethroqti
jethroqti
commented
May 30, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@psiddh
psiddh
commented
Jun 2, 2026

Copy link


Copy Markdown

Contributor
@claude Re review the latest fixes and changes
@claude
claudeBot
commented
Jun 2, 2026•

edited

Loading

Copy link


Copy Markdown

Claude finished @psiddh's task in 1m 28s —— View job
Re-review: 2-bits quantization 16a2w support (after fixes)
  •  Read PR diff and latest changes
  •  Review backends/qualcomm/builders/node_visitor.py
  •  Review backends/qualcomm/quantizer/qconfig.py
  •  Review backends/qualcomm/quantizer/quantizer.py
  •  Review backends/qualcomm/quantizer/validators.py
  •  Review backends/qualcomm/tests/test_qnn_delegate.py
  •  Post review feedback
The fix commit (c3479b8) addresses the key issues raised in the first review. Here's the assessment of the current state:
Previously Raised Issues — Status
#IssueStatus
1Per-tensor misclassification of 2-bit as 4-bitFixed — assertion added in both node_visitor.py:278-282 and validators.py:286-290 to fail fast if 2-bit reaches per-tensor path
2Asymmetric range [-2, 1] with per_tensor_symmetricUnchanged — appears intentional for the hardware
3Missing ch_axis in get_16a2w_qnn_ptq_configNot addressed — acceptable since the assertion now guards this path
4CI test failureNeeds CI run to verify
5Test loop variableFixed — now uses enumerate() correctly
Current Code Assessment
The fixes look good. The assertion-based approach in the per-tensor paths is the right call — it's better to fail loudly than to silently misclassify.
One remaining minor concern in validators.py:306-309:
elifqspec.dtype==torch.int8andqspec.quant_max-qspec.quant_min<=3:
encoding_type= (
PyQnnManager.Qnn_QuantizationEncoding_t.QNN_QUANTIZATION_ENCODING_BW_AXIS_SCALE_OFFSET
)
elifqspec.dtype==torch.int8andqspec.quant_max-qspec.quant_min<=15:
This correctly handles 2-bit in the per-channel branch, but range_ was already computed earlier in the function (line 286). The per-channel branch recomputes qspec.quant_max - qspec.quant_min inline instead of using a local variable. This is purely cosmetic — not a bug.
Summary
The PR is in good shape after the fix commit. The core implementation correctly:
  • Extends per-channel config to handle torch.int2 with range [-2, 1]
  • Applies 2-bit masking (0x03) for tensor data encoding
  • Guards the per-tensor path with clear assertions
  • Properly registers use_16a2w in QUANT_CONFIG_DICT with the per-channel annotator
  • Tests both conv2d and linear with bias/no-bias variants
Remaining action item: CI approval needed to verify test_qnn_backend_16a2w_conv2d and test_qnn_backend_16a2w_linear pass on SM8850 hardware.
View job
@psiddh
psiddh
commented
Jun 4, 2026

Copy link


Copy Markdown

Contributor
This has merge conflicts, Pls rebase and this is good to go
@psiddhpsiddh
added
the 
module: qnn
Issues related to Qualcomm's QNN delegate and code under backends/qualcomm/
label
Jun 4, 2026
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
c3479b8 to
4fddea0CompareJune 5, 2026 01:12
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@psiddh
@psiddh
psiddh
commented
Jun 5, 2026

Copy link


Copy Markdown

Contributor
@jethroqti can you check if this failure is related to the PR ? pull / test-qnn-delegate-linux / linux-job (pull_request)
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
4fddea0 to
e96a504CompareJune 5, 2026 10:36
@jethroqti
jethroqti
commented
Jun 5, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@jethroqti
jethroqti
commented
Jun 6, 2026•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
@psiddh

If I use QNN SDK version >=2.41, I did not find the "test_qnn_backend_16a2w_conv2d - AssertionError: False is not true : ref_output".

Could you try to use newer version QNN_SDK 2.41.0.251128?

Because I found HTP fixed an issue in the quantizer in v2.40.
shewu-quic
shewu-quic
reviewed
Jun 8, 2026
View reviewed changes
Comment threadbackends/qualcomm/tests/test_qnn_delegate.py
 shared_buffer=TestQNN.shared_buffer,
 )

 def test_qnn_backend_16a2w_conv2d(self):
@shewu-quicshewu-quicJun 8, 2026

Copy link


Copy Markdown

Collaborator
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add a skip condition based on qnn sdk version. Thanks
@jethroqti
Qualcomm AI Engine Direct - Support 2-bits quantization 16a2w
83e6bba
Summary:
1.Add 2-bits quantization basis 16a2w quantizer with standard symmetric
2.Support per channel and linear layers
3.Currently support soc model SM8850
Test plan:
python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_conv2d -b build-android -H ${HOST} -s ${SN} -m SM8850
python backends/qualcomm/tests/test_qnn_delegate.py TestQNNQuantizedOperator.test_qnn_backend_16a2w_linear -b build-android -H ${HOST} -s ${SN} -m SM8850
@jethroqti
jethroqtiforce-pushed
the
dev1/quant/pcq2bit
branch
from
e96a504 to
83e6bbaCompareJune 8, 2026 07:08
@jethroqti
jethroqti
commented
Jun 8, 2026

Copy link


Copy Markdown

ContributorAuthor
@pytorchbot label "release notes: qualcomm"
@psiddh
psiddh
merged commit c4e3db0
into
pytorch:mainJun 8, 2026

176 of 178 checks passed

Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@psiddhpsiddhpsiddh approved these changes
@abhinaykukkadapuabhinaykukkadapuAwaiting requested review from abhinaykukkadapuabhinaykukkadapu is a code owner

+1 more reviewer

@shewu-quicshewu-quicshewu-quic left review comments
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels

CLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.module: qnnIssues related to Qualcomm's QNN delegate and code under backends/qualcomm/release notes: qualcommChanges to the Qualcomm backend delegate

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

3 participants

@jethroqti@psiddh@shewu-quic