quantization-overview.md

Latest commit

History

History
75 lines (46 loc) · 4.07 KB

quantization-overview.md

File metadata and controls

75 lines (46 loc) · 4.07 KB
Raw

Quantization Overview

Quantization is a technique that reduces the precision of numbers used in a model’s computations and stored weights—typically from 32-bit floats to 8-bit integers. This reduces the model’s memory footprint, speeds up inference, and lowers power consumption, often with minimal loss in accuracy.

Quantization is especially important for deploying models on edge devices such as wearables, embedded systems, and microcontrollers, which often have limited compute, memory, and battery capacity. By quantizing models, we can make them significantly more efficient and suitable for these resource-constrained environments.

Quantization in ExecuTorch

ExecuTorch uses torchao as its quantization library. This integration allows ExecuTorch to leverage PyTorch-native tools for preparing, calibrating, and converting quantized models.

Quantization in ExecuTorch is backend-specific. Each backend defines how models should be quantized based on its hardware capabilities. Most ExecuTorch backends use the torchao PT2E quantization flow, which works on models exported with torch.export and enables quantization that is tailored for each backend.

The PT2E quantization workflow has three main steps:

  1. Configure a backend-specific quantizer.
  2. Prepare, calibrate, convert, and evaluate the quantized model in PyTorch
  3. Lower the model to the target backend

1. Configure a Backend-Specific Quantizer

Each backend provides its own quantizer (e.g., XNNPACKQuantizer, CoreMLQuantizer) that defines how quantization should be applied to a model in a way that is compatible with the target hardware. These quantizers usually support configs that allow users to specify quantization options such as:

  • Precision (e.g., 8-bit or 4-bit)
  • Quantization type (e.g., dynamic, static, or weight-only quantization)
  • Granularity (e.g., per-tensor, per-channel)
  • Post-Training Quantization vs. Quantization Aware Training

Not all quantization options are supported by all backends. Consult backend-specific guides for supported quantization modes and configuration, and how to initialize the backend-specific PT2E quantizer:

2. Quantize and evaluate the model

After the backend specific quantizer is defined, the PT2E quantization flow is the same for all backends. A generic example is provided below, but specific examples are given in backend documentation:

fromtorchao.quantization.pt2e.quantize_pt2eimportconvert_pt2e, prepare_pt2etraining_gm=torch.export.export(model, sample_inputs).module()
# Prepare the model for quantization using the backend-specific quantizer instanceprepared_model=prepare_pt2e(training_gm, quantizer)
# Calibrate the model on representative dataforsampleincalibration_data:
prepared_model(sample)
# Convert the calibrated model to a quantized modelquantized_model=convert_pt2e(prepared_model)

The quantized_model is a PyTorch model like any other, and can be evaluated on different tasks for accuracy. Tasks specific benchmarks are the recommended way to evaluate your quantized model, but as crude alternative you can compare to outputs with the original model using generic error metrics like SQNR:

fromtorchao.quantization.utilsimportcompute_errorout_reference=model(sample)
out_quantized=quantized_model(sample)
sqnr=compute_error(out_reference, out_quantized) # SQNR error

Note that numerics on device can differ those in PyTorch even for unquantized models, and accuracy evaluation can also be done with pybindings or on device.

3. Lower the model

The final step is to lower the quantized_model to the desired backend, as you would an unquantized one. See backend-specific pages for lowering information.

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
quantization-overview.md
Latest commit
History
History
75 lines (46 loc) · 4.07 KB
quantization-overview.md
File metadata and controls
75 lines (46 loc) · 4.07 KB
Raw
Quantization Overview
Quantization is a technique that reduces the precision of numbers used in a model’s computations and stored weights—typically from 32-bit floats to 8-bit integers. This reduces the model’s memory footprint, speeds up inference, and lowers power consumption, often with minimal loss in accuracy.
Quantization is especially important for deploying models on edge devices such as wearables, embedded systems, and microcontrollers, which often have limited compute, memory, and battery capacity. By quantizing models, we can make them significantly more efficient and suitable for these resource-constrained environments.
Quantization in ExecuTorch
ExecuTorch uses torchao as its quantization library. This integration allows ExecuTorch to leverage PyTorch-native tools for preparing, calibrating, and converting quantized models.
Quantization in ExecuTorch is backend-specific. Each backend defines how models should be quantized based on its hardware capabilities. Most ExecuTorch backends use the torchao PT2E quantization flow, which works on models exported with torch.export and enables quantization that is tailored for each backend.
The PT2E quantization workflow has three main steps:
  1. Configure a backend-specific quantizer.
  2. Prepare, calibrate, convert, and evaluate the quantized model in PyTorch
  3. Lower the model to the target backend
1. Configure a Backend-Specific Quantizer
Each backend provides its own quantizer (e.g., XNNPACKQuantizer, CoreMLQuantizer) that defines how quantization should be applied to a model in a way that is compatible with the target hardware.
These quantizers usually support configs that allow users to specify quantization options such as:
  • Precision (e.g., 8-bit or 4-bit)
  • Quantization type (e.g., dynamic, static, or weight-only quantization)
  • Granularity (e.g., per-tensor, per-channel)
  • Post-Training Quantization vs. Quantization Aware Training
Not all quantization options are supported by all backends. Consult backend-specific guides for supported quantization modes and configuration, and how to initialize the backend-specific PT2E quantizer:
2. Quantize and evaluate the model
After the backend specific quantizer is defined, the PT2E quantization flow is the same for all backends. A generic example is provided below, but specific examples are given in backend documentation:
fromtorchao.quantization.pt2e.quantize_pt2eimportconvert_pt2e, prepare_pt2etraining_gm=torch.export.export(model, sample_inputs).module()
# Prepare the model for quantization using the backend-specific quantizer instanceprepared_model=prepare_pt2e(training_gm, quantizer)
# Calibrate the model on representative dataforsampleincalibration_data:
prepared_model(sample)
# Convert the calibrated model to a quantized modelquantized_model=convert_pt2e(prepared_model)
The quantized_model is a PyTorch model like any other, and can be evaluated on different tasks for accuracy.
Tasks specific benchmarks are the recommended way to evaluate your quantized model, but as crude alternative you can compare to outputs with the original model using generic error metrics like SQNR:
fromtorchao.quantization.utilsimportcompute_errorout_reference=model(sample)
out_quantized=quantized_model(sample)
sqnr=compute_error(out_reference, out_quantized) # SQNR error
Note that numerics on device can differ those in PyTorch even for unquantized models, and accuracy evaluation can also be done with pybindings or on device.
3. Lower the model
The final step is to lower the quantized_model to the desired backend, as you would an unquantized one. See backend-specific pages for lowering information.

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
quantization-overview.md
Latest commit
History
History
75 lines (46 loc) · 4.07 KB
quantization-overview.md
File metadata and controls
75 lines (46 loc) · 4.07 KB
Raw
Quantization Overview
Quantization is a technique that reduces the precision of numbers used in a model’s computations and stored weights—typically from 32-bit floats to 8-bit integers. This reduces the model’s memory footprint, speeds up inference, and lowers power consumption, often with minimal loss in accuracy.
Quantization is especially important for deploying models on edge devices such as wearables, embedded systems, and microcontrollers, which often have limited compute, memory, and battery capacity. By quantizing models, we can make them significantly more efficient and suitable for these resource-constrained environments.
Quantization in ExecuTorch
ExecuTorch uses torchao as its quantization library. This integration allows ExecuTorch to leverage PyTorch-native tools for preparing, calibrating, and converting quantized models.
Quantization in ExecuTorch is backend-specific. Each backend defines how models should be quantized based on its hardware capabilities. Most ExecuTorch backends use the torchao PT2E quantization flow, which works on models exported with torch.export and enables quantization that is tailored for each backend.
The PT2E quantization workflow has three main steps:
  1. Configure a backend-specific quantizer.
  2. Prepare, calibrate, convert, and evaluate the quantized model in PyTorch
  3. Lower the model to the target backend
1. Configure a Backend-Specific Quantizer
Each backend provides its own quantizer (e.g., XNNPACKQuantizer, CoreMLQuantizer) that defines how quantization should be applied to a model in a way that is compatible with the target hardware.
These quantizers usually support configs that allow users to specify quantization options such as:
  • Precision (e.g., 8-bit or 4-bit)
  • Quantization type (e.g., dynamic, static, or weight-only quantization)
  • Granularity (e.g., per-tensor, per-channel)
  • Post-Training Quantization vs. Quantization Aware Training
Not all quantization options are supported by all backends. Consult backend-specific guides for supported quantization modes and configuration, and how to initialize the backend-specific PT2E quantizer:
2. Quantize and evaluate the model
After the backend specific quantizer is defined, the PT2E quantization flow is the same for all backends. A generic example is provided below, but specific examples are given in backend documentation:
fromtorchao.quantization.pt2e.quantize_pt2eimportconvert_pt2e, prepare_pt2etraining_gm=torch.export.export(model, sample_inputs).module()
# Prepare the model for quantization using the backend-specific quantizer instanceprepared_model=prepare_pt2e(training_gm, quantizer)
# Calibrate the model on representative dataforsampleincalibration_data:
prepared_model(sample)
# Convert the calibrated model to a quantized modelquantized_model=convert_pt2e(prepared_model)
The quantized_model is a PyTorch model like any other, and can be evaluated on different tasks for accuracy.
Tasks specific benchmarks are the recommended way to evaluate your quantized model, but as crude alternative you can compare to outputs with the original model using generic error metrics like SQNR:
fromtorchao.quantization.utilsimportcompute_errorout_reference=model(sample)
out_quantized=quantized_model(sample)
sqnr=compute_error(out_reference, out_quantized) # SQNR error
Note that numerics on device can differ those in PyTorch even for unquantized models, and accuracy evaluation can also be done with pybindings or on device.
3. Lower the model
The final step is to lower the quantized_model to the desired backend, as you would an unquantized one. See backend-specific pages for lowering information.

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
quantization-overview.md
Latest commit
History
History
75 lines (46 loc) · 4.07 KB
quantization-overview.md
File metadata and controls
75 lines (46 loc) · 4.07 KB
Raw
Quantization Overview
Quantization is a technique that reduces the precision of numbers used in a model’s computations and stored weights—typically from 32-bit floats to 8-bit integers. This reduces the model’s memory footprint, speeds up inference, and lowers power consumption, often with minimal loss in accuracy.
Quantization is especially important for deploying models on edge devices such as wearables, embedded systems, and microcontrollers, which often have limited compute, memory, and battery capacity. By quantizing models, we can make them significantly more efficient and suitable for these resource-constrained environments.
Quantization in ExecuTorch
ExecuTorch uses torchao as its quantization library. This integration allows ExecuTorch to leverage PyTorch-native tools for preparing, calibrating, and converting quantized models.
Quantization in ExecuTorch is backend-specific. Each backend defines how models should be quantized based on its hardware capabilities. Most ExecuTorch backends use the torchao PT2E quantization flow, which works on models exported with torch.export and enables quantization that is tailored for each backend.
The PT2E quantization workflow has three main steps:
  1. Configure a backend-specific quantizer.
  2. Prepare, calibrate, convert, and evaluate the quantized model in PyTorch
  3. Lower the model to the target backend
1. Configure a Backend-Specific Quantizer
Each backend provides its own quantizer (e.g., XNNPACKQuantizer, CoreMLQuantizer) that defines how quantization should be applied to a model in a way that is compatible with the target hardware.
These quantizers usually support configs that allow users to specify quantization options such as:
  • Precision (e.g., 8-bit or 4-bit)
  • Quantization type (e.g., dynamic, static, or weight-only quantization)
  • Granularity (e.g., per-tensor, per-channel)
  • Post-Training Quantization vs. Quantization Aware Training
Not all quantization options are supported by all backends. Consult backend-specific guides for supported quantization modes and configuration, and how to initialize the backend-specific PT2E quantizer:
2. Quantize and evaluate the model
After the backend specific quantizer is defined, the PT2E quantization flow is the same for all backends. A generic example is provided below, but specific examples are given in backend documentation:
fromtorchao.quantization.pt2e.quantize_pt2eimportconvert_pt2e, prepare_pt2etraining_gm=torch.export.export(model, sample_inputs).module()
# Prepare the model for quantization using the backend-specific quantizer instanceprepared_model=prepare_pt2e(training_gm, quantizer)
# Calibrate the model on representative dataforsampleincalibration_data:
prepared_model(sample)
# Convert the calibrated model to a quantized modelquantized_model=convert_pt2e(prepared_model)
The quantized_model is a PyTorch model like any other, and can be evaluated on different tasks for accuracy.
Tasks specific benchmarks are the recommended way to evaluate your quantized model, but as crude alternative you can compare to outputs with the original model using generic error metrics like SQNR:
fromtorchao.quantization.utilsimportcompute_errorout_reference=model(sample)
out_quantized=quantized_model(sample)
sqnr=compute_error(out_reference, out_quantized) # SQNR error
Note that numerics on device can differ those in PyTorch even for unquantized models, and accuracy evaluation can also be done with pybindings or on device.
3. Lower the model
The final step is to lower the quantized_model to the desired backend, as you would an unquantized one. See backend-specific pages for lowering information.

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
quantization-overview.md
Latest commit
History
History
75 lines (46 loc) · 4.07 KB
quantization-overview.md
File metadata and controls
75 lines (46 loc) · 4.07 KB
Raw
Quantization Overview
Quantization is a technique that reduces the precision of numbers used in a model’s computations and stored weights—typically from 32-bit floats to 8-bit integers. This reduces the model’s memory footprint, speeds up inference, and lowers power consumption, often with minimal loss in accuracy.
Quantization is especially important for deploying models on edge devices such as wearables, embedded systems, and microcontrollers, which often have limited compute, memory, and battery capacity. By quantizing models, we can make them significantly more efficient and suitable for these resource-constrained environments.
Quantization in ExecuTorch
ExecuTorch uses torchao as its quantization library. This integration allows ExecuTorch to leverage PyTorch-native tools for preparing, calibrating, and converting quantized models.
Quantization in ExecuTorch is backend-specific. Each backend defines how models should be quantized based on its hardware capabilities. Most ExecuTorch backends use the torchao PT2E quantization flow, which works on models exported with torch.export and enables quantization that is tailored for each backend.
The PT2E quantization workflow has three main steps:
  1. Configure a backend-specific quantizer.
  2. Prepare, calibrate, convert, and evaluate the quantized model in PyTorch
  3. Lower the model to the target backend
1. Configure a Backend-Specific Quantizer
Each backend provides its own quantizer (e.g., XNNPACKQuantizer, CoreMLQuantizer) that defines how quantization should be applied to a model in a way that is compatible with the target hardware.
These quantizers usually support configs that allow users to specify quantization options such as:
  • Precision (e.g., 8-bit or 4-bit)
  • Quantization type (e.g., dynamic, static, or weight-only quantization)
  • Granularity (e.g., per-tensor, per-channel)
  • Post-Training Quantization vs. Quantization Aware Training
Not all quantization options are supported by all backends. Consult backend-specific guides for supported quantization modes and configuration, and how to initialize the backend-specific PT2E quantizer:
2. Quantize and evaluate the model
After the backend specific quantizer is defined, the PT2E quantization flow is the same for all backends. A generic example is provided below, but specific examples are given in backend documentation:
fromtorchao.quantization.pt2e.quantize_pt2eimportconvert_pt2e, prepare_pt2etraining_gm=torch.export.export(model, sample_inputs).module()
# Prepare the model for quantization using the backend-specific quantizer instanceprepared_model=prepare_pt2e(training_gm, quantizer)
# Calibrate the model on representative dataforsampleincalibration_data:
prepared_model(sample)
# Convert the calibrated model to a quantized modelquantized_model=convert_pt2e(prepared_model)
The quantized_model is a PyTorch model like any other, and can be evaluated on different tasks for accuracy.
Tasks specific benchmarks are the recommended way to evaluate your quantized model, but as crude alternative you can compare to outputs with the original model using generic error metrics like SQNR:
fromtorchao.quantization.utilsimportcompute_errorout_reference=model(sample)
out_quantized=quantized_model(sample)
sqnr=compute_error(out_reference, out_quantized) # SQNR error
Note that numerics on device can differ those in PyTorch even for unquantized models, and accuracy evaluation can also be done with pybindings or on device.
3. Lower the model
The final step is to lower the quantized_model to the desired backend, as you would an unquantized one. See backend-specific pages for lowering information.

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
quantization-overview.md
Latest commit
History
History
75 lines (46 loc) · 4.07 KB
quantization-overview.md
File metadata and controls
75 lines (46 loc) · 4.07 KB
Raw
Quantization Overview
Quantization is a technique that reduces the precision of numbers used in a model’s computations and stored weights—typically from 32-bit floats to 8-bit integers. This reduces the model’s memory footprint, speeds up inference, and lowers power consumption, often with minimal loss in accuracy.
Quantization is especially important for deploying models on edge devices such as wearables, embedded systems, and microcontrollers, which often have limited compute, memory, and battery capacity. By quantizing models, we can make them significantly more efficient and suitable for these resource-constrained environments.
Quantization in ExecuTorch
ExecuTorch uses torchao as its quantization library. This integration allows ExecuTorch to leverage PyTorch-native tools for preparing, calibrating, and converting quantized models.
Quantization in ExecuTorch is backend-specific. Each backend defines how models should be quantized based on its hardware capabilities. Most ExecuTorch backends use the torchao PT2E quantization flow, which works on models exported with torch.export and enables quantization that is tailored for each backend.
The PT2E quantization workflow has three main steps:
  1. Configure a backend-specific quantizer.
  2. Prepare, calibrate, convert, and evaluate the quantized model in PyTorch
  3. Lower the model to the target backend
1. Configure a Backend-Specific Quantizer
Each backend provides its own quantizer (e.g., XNNPACKQuantizer, CoreMLQuantizer) that defines how quantization should be applied to a model in a way that is compatible with the target hardware.
These quantizers usually support configs that allow users to specify quantization options such as:
  • Precision (e.g., 8-bit or 4-bit)
  • Quantization type (e.g., dynamic, static, or weight-only quantization)
  • Granularity (e.g., per-tensor, per-channel)
  • Post-Training Quantization vs. Quantization Aware Training
Not all quantization options are supported by all backends. Consult backend-specific guides for supported quantization modes and configuration, and how to initialize the backend-specific PT2E quantizer:
2. Quantize and evaluate the model
After the backend specific quantizer is defined, the PT2E quantization flow is the same for all backends. A generic example is provided below, but specific examples are given in backend documentation:
fromtorchao.quantization.pt2e.quantize_pt2eimportconvert_pt2e, prepare_pt2etraining_gm=torch.export.export(model, sample_inputs).module()
# Prepare the model for quantization using the backend-specific quantizer instanceprepared_model=prepare_pt2e(training_gm, quantizer)
# Calibrate the model on representative dataforsampleincalibration_data:
prepared_model(sample)
# Convert the calibrated model to a quantized modelquantized_model=convert_pt2e(prepared_model)
The quantized_model is a PyTorch model like any other, and can be evaluated on different tasks for accuracy.
Tasks specific benchmarks are the recommended way to evaluate your quantized model, but as crude alternative you can compare to outputs with the original model using generic error metrics like SQNR:
fromtorchao.quantization.utilsimportcompute_errorout_reference=model(sample)
out_quantized=quantized_model(sample)
sqnr=compute_error(out_reference, out_quantized) # SQNR error
Note that numerics on device can differ those in PyTorch even for unquantized models, and accuracy evaluation can also be done with pybindings or on device.
3. Lower the model
The final step is to lower the quantized_model to the desired backend, as you would an unquantized one. See backend-specific pages for lowering information.

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
quantization-overview.md
Latest commit
History
History
75 lines (46 loc) · 4.07 KB
quantization-overview.md
File metadata and controls
75 lines (46 loc) · 4.07 KB
Raw
Quantization Overview
Quantization is a technique that reduces the precision of numbers used in a model’s computations and stored weights—typically from 32-bit floats to 8-bit integers. This reduces the model’s memory footprint, speeds up inference, and lowers power consumption, often with minimal loss in accuracy.
Quantization is especially important for deploying models on edge devices such as wearables, embedded systems, and microcontrollers, which often have limited compute, memory, and battery capacity. By quantizing models, we can make them significantly more efficient and suitable for these resource-constrained environments.
Quantization in ExecuTorch
ExecuTorch uses torchao as its quantization library. This integration allows ExecuTorch to leverage PyTorch-native tools for preparing, calibrating, and converting quantized models.
Quantization in ExecuTorch is backend-specific. Each backend defines how models should be quantized based on its hardware capabilities. Most ExecuTorch backends use the torchao PT2E quantization flow, which works on models exported with torch.export and enables quantization that is tailored for each backend.
The PT2E quantization workflow has three main steps:
  1. Configure a backend-specific quantizer.
  2. Prepare, calibrate, convert, and evaluate the quantized model in PyTorch
  3. Lower the model to the target backend
1. Configure a Backend-Specific Quantizer
Each backend provides its own quantizer (e.g., XNNPACKQuantizer, CoreMLQuantizer) that defines how quantization should be applied to a model in a way that is compatible with the target hardware.
These quantizers usually support configs that allow users to specify quantization options such as:
  • Precision (e.g., 8-bit or 4-bit)
  • Quantization type (e.g., dynamic, static, or weight-only quantization)
  • Granularity (e.g., per-tensor, per-channel)
  • Post-Training Quantization vs. Quantization Aware Training
Not all quantization options are supported by all backends. Consult backend-specific guides for supported quantization modes and configuration, and how to initialize the backend-specific PT2E quantizer:
2. Quantize and evaluate the model
After the backend specific quantizer is defined, the PT2E quantization flow is the same for all backends. A generic example is provided below, but specific examples are given in backend documentation:
fromtorchao.quantization.pt2e.quantize_pt2eimportconvert_pt2e, prepare_pt2etraining_gm=torch.export.export(model, sample_inputs).module()
# Prepare the model for quantization using the backend-specific quantizer instanceprepared_model=prepare_pt2e(training_gm, quantizer)
# Calibrate the model on representative dataforsampleincalibration_data:
prepared_model(sample)
# Convert the calibrated model to a quantized modelquantized_model=convert_pt2e(prepared_model)
The quantized_model is a PyTorch model like any other, and can be evaluated on different tasks for accuracy.
Tasks specific benchmarks are the recommended way to evaluate your quantized model, but as crude alternative you can compare to outputs with the original model using generic error metrics like SQNR:
fromtorchao.quantization.utilsimportcompute_errorout_reference=model(sample)
out_quantized=quantized_model(sample)
sqnr=compute_error(out_reference, out_quantized) # SQNR error
Note that numerics on device can differ those in PyTorch even for unquantized models, and accuracy evaluation can also be done with pybindings or on device.
3. Lower the model
The final step is to lower the quantized_model to the desired backend, as you would an unquantized one. See backend-specific pages for lowering information.

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
quantization-overview.md
Latest commit
History
History
75 lines (46 loc) · 4.07 KB
quantization-overview.md
File metadata and controls
75 lines (46 loc) · 4.07 KB
Raw
Quantization Overview
Quantization is a technique that reduces the precision of numbers used in a model’s computations and stored weights—typically from 32-bit floats to 8-bit integers. This reduces the model’s memory footprint, speeds up inference, and lowers power consumption, often with minimal loss in accuracy.
Quantization is especially important for deploying models on edge devices such as wearables, embedded systems, and microcontrollers, which often have limited compute, memory, and battery capacity. By quantizing models, we can make them significantly more efficient and suitable for these resource-constrained environments.
Quantization in ExecuTorch
ExecuTorch uses torchao as its quantization library. This integration allows ExecuTorch to leverage PyTorch-native tools for preparing, calibrating, and converting quantized models.
Quantization in ExecuTorch is backend-specific. Each backend defines how models should be quantized based on its hardware capabilities. Most ExecuTorch backends use the torchao PT2E quantization flow, which works on models exported with torch.export and enables quantization that is tailored for each backend.
The PT2E quantization workflow has three main steps:
  1. Configure a backend-specific quantizer.
  2. Prepare, calibrate, convert, and evaluate the quantized model in PyTorch
  3. Lower the model to the target backend
1. Configure a Backend-Specific Quantizer
Each backend provides its own quantizer (e.g., XNNPACKQuantizer, CoreMLQuantizer) that defines how quantization should be applied to a model in a way that is compatible with the target hardware.
These quantizers usually support configs that allow users to specify quantization options such as:
  • Precision (e.g., 8-bit or 4-bit)
  • Quantization type (e.g., dynamic, static, or weight-only quantization)
  • Granularity (e.g., per-tensor, per-channel)
  • Post-Training Quantization vs. Quantization Aware Training
Not all quantization options are supported by all backends. Consult backend-specific guides for supported quantization modes and configuration, and how to initialize the backend-specific PT2E quantizer:
2. Quantize and evaluate the model
After the backend specific quantizer is defined, the PT2E quantization flow is the same for all backends. A generic example is provided below, but specific examples are given in backend documentation:
fromtorchao.quantization.pt2e.quantize_pt2eimportconvert_pt2e, prepare_pt2etraining_gm=torch.export.export(model, sample_inputs).module()
# Prepare the model for quantization using the backend-specific quantizer instanceprepared_model=prepare_pt2e(training_gm, quantizer)
# Calibrate the model on representative dataforsampleincalibration_data:
prepared_model(sample)
# Convert the calibrated model to a quantized modelquantized_model=convert_pt2e(prepared_model)
The quantized_model is a PyTorch model like any other, and can be evaluated on different tasks for accuracy.
Tasks specific benchmarks are the recommended way to evaluate your quantized model, but as crude alternative you can compare to outputs with the original model using generic error metrics like SQNR:
fromtorchao.quantization.utilsimportcompute_errorout_reference=model(sample)
out_quantized=quantized_model(sample)
sqnr=compute_error(out_reference, out_quantized) # SQNR error
Note that numerics on device can differ those in PyTorch even for unquantized models, and accuracy evaluation can also be done with pybindings or on device.
3. Lower the model
The final step is to lower the quantized_model to the desired backend, as you would an unquantized one. See backend-specific pages for lowering information.