[TOPI,x86] Improve performance on int8 conv2d on x86 - #9966

Merged
masahi merged 1 commit into
apache:mainfrom
tkonolige:faster_int8_conv
Jan 19, 2022
Merged

[TOPI,x86] Improve performance on int8 conv2d on x86#9966
masahi merged 1 commit into
apache:mainfrom
tkonolige:faster_int8_conv

Conversation

@tkonolige

@tkonoligetkonolige commented Jan 18, 2022•
edited
Loading

Copy link
Copy Markdown
Contributor

Appended fused operations in cov2d for int8 were computed in a separate loop from the main conv2d computation:

for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for j in ..
out = out + fused subsequent ops

This patch moves the fused ops one more loop nesting inwards to get

for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops

On quantized mobilenetv2, this results in approximately a 30% speedup.

@masahi@mbrookhart

Tristan Konolige
[TOPI,x86] Improve performance on int8 conv2d on x86
858b30c
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
masahi
masahi approved these changes Jan 18, 2022
View reviewed changes

@masahimasahi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense, and good speed up!

Curious how much it helps for other models. At first I thought this would be a bigger win for larger workloads (with corresponding larger write cache). It is easy to test quantized resnet50, inception v3, or mobilenet v3 via PyTorch.

mbrookhart
mbrookhart approved these changes Jan 18, 2022
View reviewed changes

@mbrookhartmbrookhart left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay!

@tkonolige

tkonolige commented Jan 18, 2022•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor
ModelBefore (ms)After (ms)
resnet505.52824.7501
inception_v33.43203.2994
mobilenetv21.52391.1931

@masahi
masahi merged commit 19717aa into apache:mainJan 19, 2022
yuanfz98 pushed a commit to yuanfz98/tvm that referenced this pull request Jan 24, 2022
Tristan Konolige@yuanfz98
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
ed65270
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
crazydemo pushed a commit to crazydemo/tvm that referenced this pull request Jan 27, 2022
Tristan KonoligeIvy
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
21351bb
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
ylc pushed a commit to ylc/tvm that referenced this pull request Feb 16, 2022
Tristan Konolige@ylc
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
b08f596
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@masahimasahimasahi approved these changes
@mbrookhartmbrookhartmbrookhart approved these changes
@HuyuweiHuyuweiAwaiting requested review from Huyuwei
@jcf94jcf94Awaiting requested review from jcf94
@jwfrommjwfrommAwaiting requested review from jwfromm
@kevinthesunkevinthesunAwaiting requested review from kevinthesun
@LaurawlyLaurawlyAwaiting requested review from Laurawly
@vinx13vinx13Awaiting requested review from vinx13
@yzhliuyzhliuAwaiting requested review from yzhliu
@ZihengJiangZihengJiangAwaiting requested review from ZihengJiang
+1 more reviewer
@michalpiszczekmichalpiszczekmichalpiszczek approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tkonolige@masahi@mbrookhart@michalpiszczek
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
[TOPI,x86] Improve performance on int8 conv2d on x86 - #9966
Merged
masahi merged 1 commit into
apache:mainfrom 
tkonolige:faster_int8_conv
Jan 19, 2022
Merged
[TOPI,x86] Improve performance on int8 conv2d on x86#9966
masahi merged 1 commit into
apache:mainfrom 
tkonolige:faster_int8_conv
Conversation
@tkonolige
@tkonoligetkonolige
commented
Jan 18, 2022•

edited

Loading

Copy link


Copy Markdown

Contributor
Appended fused operations in cov2d for int8 were computed in a separate loop from the main conv2d computation:
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for j in ..
out = out + fused subsequent ops

This patch moves the fused ops one more loop nesting inwards to get
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops

On quantized mobilenetv2, this results in approximately a 30% speedup.
@masahi@mbrookhart
Tristan Konolige
[TOPI,x86] Improve performance on int8 conv2d on x86
858b30c
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
masahi
masahi
approved these changes
Jan 18, 2022
View reviewed changes
@masahimasahi
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense, and good speed up!
Curious how much it helps for other models. At first I thought this would be a bigger win for larger workloads (with corresponding larger write cache). It is easy to test quantized resnet50, inception v3, or mobilenet v3 via PyTorch.
mbrookhart
mbrookhart
approved these changes
Jan 18, 2022
View reviewed changes
@mbrookhartmbrookhart
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay!
@tkonolige
tkonolige
commented
Jan 18, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
ModelBefore (ms)After (ms)
resnet505.52824.7501
inception_v33.43203.2994
mobilenetv21.52391.1931
@masahi
masahi
merged commit 19717aa
into
apache:mainJan 19, 2022
yuanfz98 pushed a commit
to yuanfz98/tvm
that referenced
this pull request
Jan 24, 2022
Tristan Konolige@yuanfz98
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
ed65270
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
crazydemo pushed a commit
to crazydemo/tvm
that referenced
this pull request
Jan 27, 2022
Tristan KonoligeIvy
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
21351bb
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
ylc pushed a commit
to ylc/tvm
that referenced
this pull request
Feb 16, 2022
Tristan Konolige@ylc
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
b08f596
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@masahimasahimasahi approved these changes
@mbrookhartmbrookhartmbrookhart approved these changes
@HuyuweiHuyuweiAwaiting requested review from Huyuwei
@jcf94jcf94Awaiting requested review from jcf94
@jwfrommjwfrommAwaiting requested review from jwfromm
@kevinthesunkevinthesunAwaiting requested review from kevinthesun
@LaurawlyLaurawlyAwaiting requested review from Laurawly
@vinx13vinx13Awaiting requested review from vinx13
@yzhliuyzhliuAwaiting requested review from yzhliu
@ZihengJiangZihengJiangAwaiting requested review from ZihengJiang

+1 more reviewer

@michalpiszczekmichalpiszczekmichalpiszczek approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@tkonolige@masahi@mbrookhart@michalpiszczek

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[TOPI,x86] Improve performance on int8 conv2d on x86 - #9966
Merged
masahi merged 1 commit into
apache:mainfrom 
tkonolige:faster_int8_conv
Jan 19, 2022
Merged
[TOPI,x86] Improve performance on int8 conv2d on x86#9966
masahi merged 1 commit into
apache:mainfrom 
tkonolige:faster_int8_conv
Conversation
@tkonolige
@tkonoligetkonolige
commented
Jan 18, 2022•

edited

Loading

Copy link


Copy Markdown

Contributor
Appended fused operations in cov2d for int8 were computed in a separate loop from the main conv2d computation:
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for j in ..
out = out + fused subsequent ops

This patch moves the fused ops one more loop nesting inwards to get
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops

On quantized mobilenetv2, this results in approximately a 30% speedup.
@masahi@mbrookhart
Tristan Konolige
[TOPI,x86] Improve performance on int8 conv2d on x86
858b30c
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
masahi
masahi
approved these changes
Jan 18, 2022
View reviewed changes
@masahimasahi
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense, and good speed up!
Curious how much it helps for other models. At first I thought this would be a bigger win for larger workloads (with corresponding larger write cache). It is easy to test quantized resnet50, inception v3, or mobilenet v3 via PyTorch.
mbrookhart
mbrookhart
approved these changes
Jan 18, 2022
View reviewed changes
@mbrookhartmbrookhart
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay!
@tkonolige
tkonolige
commented
Jan 18, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
ModelBefore (ms)After (ms)
resnet505.52824.7501
inception_v33.43203.2994
mobilenetv21.52391.1931
@masahi
masahi
merged commit 19717aa
into
apache:mainJan 19, 2022
yuanfz98 pushed a commit
to yuanfz98/tvm
that referenced
this pull request
Jan 24, 2022
Tristan Konolige@yuanfz98
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
ed65270
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
crazydemo pushed a commit
to crazydemo/tvm
that referenced
this pull request
Jan 27, 2022
Tristan KonoligeIvy
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
21351bb
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
ylc pushed a commit
to ylc/tvm
that referenced
this pull request
Feb 16, 2022
Tristan Konolige@ylc
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
b08f596
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@masahimasahimasahi approved these changes
@mbrookhartmbrookhartmbrookhart approved these changes
@HuyuweiHuyuweiAwaiting requested review from Huyuwei
@jcf94jcf94Awaiting requested review from jcf94
@jwfrommjwfrommAwaiting requested review from jwfromm
@kevinthesunkevinthesunAwaiting requested review from kevinthesun
@LaurawlyLaurawlyAwaiting requested review from Laurawly
@vinx13vinx13Awaiting requested review from vinx13
@yzhliuyzhliuAwaiting requested review from yzhliu
@ZihengJiangZihengJiangAwaiting requested review from ZihengJiang

+1 more reviewer

@michalpiszczekmichalpiszczekmichalpiszczek approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@tkonolige@masahi@mbrookhart@michalpiszczek

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[TOPI,x86] Improve performance on int8 conv2d on x86 - #9966
Merged
masahi merged 1 commit into
apache:mainfrom 
tkonolige:faster_int8_conv
Jan 19, 2022
Merged
[TOPI,x86] Improve performance on int8 conv2d on x86#9966
masahi merged 1 commit into
apache:mainfrom 
tkonolige:faster_int8_conv
Conversation
@tkonolige
@tkonoligetkonolige
commented
Jan 18, 2022•

edited

Loading

Copy link


Copy Markdown

Contributor
Appended fused operations in cov2d for int8 were computed in a separate loop from the main conv2d computation:
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for j in ..
out = out + fused subsequent ops

This patch moves the fused ops one more loop nesting inwards to get
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops

On quantized mobilenetv2, this results in approximately a 30% speedup.
@masahi@mbrookhart
Tristan Konolige
[TOPI,x86] Improve performance on int8 conv2d on x86
858b30c
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
masahi
masahi
approved these changes
Jan 18, 2022
View reviewed changes
@masahimasahi
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense, and good speed up!
Curious how much it helps for other models. At first I thought this would be a bigger win for larger workloads (with corresponding larger write cache). It is easy to test quantized resnet50, inception v3, or mobilenet v3 via PyTorch.
mbrookhart
mbrookhart
approved these changes
Jan 18, 2022
View reviewed changes
@mbrookhartmbrookhart
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay!
@tkonolige
tkonolige
commented
Jan 18, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
ModelBefore (ms)After (ms)
resnet505.52824.7501
inception_v33.43203.2994
mobilenetv21.52391.1931
@masahi
masahi
merged commit 19717aa
into
apache:mainJan 19, 2022
yuanfz98 pushed a commit
to yuanfz98/tvm
that referenced
this pull request
Jan 24, 2022
Tristan Konolige@yuanfz98
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
ed65270
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
crazydemo pushed a commit
to crazydemo/tvm
that referenced
this pull request
Jan 27, 2022
Tristan KonoligeIvy
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
21351bb
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
ylc pushed a commit
to ylc/tvm
that referenced
this pull request
Feb 16, 2022
Tristan Konolige@ylc
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
b08f596
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@masahimasahimasahi approved these changes
@mbrookhartmbrookhartmbrookhart approved these changes
@HuyuweiHuyuweiAwaiting requested review from Huyuwei
@jcf94jcf94Awaiting requested review from jcf94
@jwfrommjwfrommAwaiting requested review from jwfromm
@kevinthesunkevinthesunAwaiting requested review from kevinthesun
@LaurawlyLaurawlyAwaiting requested review from Laurawly
@vinx13vinx13Awaiting requested review from vinx13
@yzhliuyzhliuAwaiting requested review from yzhliu
@ZihengJiangZihengJiangAwaiting requested review from ZihengJiang

+1 more reviewer

@michalpiszczekmichalpiszczekmichalpiszczek approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@tkonolige@masahi@mbrookhart@michalpiszczek

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
[TOPI,x86] Improve performance on int8 conv2d on x86 - #9966
Merged
masahi merged 1 commit into
apache:mainfrom 
tkonolige:faster_int8_conv
Jan 19, 2022
Merged
[TOPI,x86] Improve performance on int8 conv2d on x86#9966
masahi merged 1 commit into
apache:mainfrom 
tkonolige:faster_int8_conv
Conversation
@tkonolige
@tkonoligetkonolige
commented
Jan 18, 2022•

edited

Loading

Copy link


Copy Markdown

Contributor
Appended fused operations in cov2d for int8 were computed in a separate loop from the main conv2d computation:
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for j in ..
out = out + fused subsequent ops

This patch moves the fused ops one more loop nesting inwards to get
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops

On quantized mobilenetv2, this results in approximately a 30% speedup.
@masahi@mbrookhart
Tristan Konolige
[TOPI,x86] Improve performance on int8 conv2d on x86
858b30c
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
masahi
masahi
approved these changes
Jan 18, 2022
View reviewed changes
@masahimasahi
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense, and good speed up!
Curious how much it helps for other models. At first I thought this would be a bigger win for larger workloads (with corresponding larger write cache). It is easy to test quantized resnet50, inception v3, or mobilenet v3 via PyTorch.
mbrookhart
mbrookhart
approved these changes
Jan 18, 2022
View reviewed changes
@mbrookhartmbrookhart
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay!
@tkonolige
tkonolige
commented
Jan 18, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
ModelBefore (ms)After (ms)
resnet505.52824.7501
inception_v33.43203.2994
mobilenetv21.52391.1931
@masahi
masahi
merged commit 19717aa
into
apache:mainJan 19, 2022
yuanfz98 pushed a commit
to yuanfz98/tvm
that referenced
this pull request
Jan 24, 2022
Tristan Konolige@yuanfz98
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
ed65270
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
crazydemo pushed a commit
to crazydemo/tvm
that referenced
this pull request
Jan 27, 2022
Tristan KonoligeIvy
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
21351bb
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
ylc pushed a commit
to ylc/tvm
that referenced
this pull request
Feb 16, 2022
Tristan Konolige@ylc
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
b08f596
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@masahimasahimasahi approved these changes
@mbrookhartmbrookhartmbrookhart approved these changes
@HuyuweiHuyuweiAwaiting requested review from Huyuwei
@jcf94jcf94Awaiting requested review from jcf94
@jwfrommjwfrommAwaiting requested review from jwfromm
@kevinthesunkevinthesunAwaiting requested review from kevinthesun
@LaurawlyLaurawlyAwaiting requested review from Laurawly
@vinx13vinx13Awaiting requested review from vinx13
@yzhliuyzhliuAwaiting requested review from yzhliu
@ZihengJiangZihengJiangAwaiting requested review from ZihengJiang

+1 more reviewer

@michalpiszczekmichalpiszczekmichalpiszczek approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@tkonolige@masahi@mbrookhart@michalpiszczek

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[TOPI,x86] Improve performance on int8 conv2d on x86 - #9966
Merged
masahi merged 1 commit into
apache:mainfrom 
tkonolige:faster_int8_conv
Jan 19, 2022
Merged
[TOPI,x86] Improve performance on int8 conv2d on x86#9966
masahi merged 1 commit into
apache:mainfrom 
tkonolige:faster_int8_conv
Conversation
@tkonolige
@tkonoligetkonolige
commented
Jan 18, 2022•

edited

Loading

Copy link


Copy Markdown

Contributor
Appended fused operations in cov2d for int8 were computed in a separate loop from the main conv2d computation:
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for j in ..
out = out + fused subsequent ops

This patch moves the fused ops one more loop nesting inwards to get
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops

On quantized mobilenetv2, this results in approximately a 30% speedup.
@masahi@mbrookhart
Tristan Konolige
[TOPI,x86] Improve performance on int8 conv2d on x86
858b30c
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
masahi
masahi
approved these changes
Jan 18, 2022
View reviewed changes
@masahimasahi
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense, and good speed up!
Curious how much it helps for other models. At first I thought this would be a bigger win for larger workloads (with corresponding larger write cache). It is easy to test quantized resnet50, inception v3, or mobilenet v3 via PyTorch.
mbrookhart
mbrookhart
approved these changes
Jan 18, 2022
View reviewed changes
@mbrookhartmbrookhart
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay!
@tkonolige
tkonolige
commented
Jan 18, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
ModelBefore (ms)After (ms)
resnet505.52824.7501
inception_v33.43203.2994
mobilenetv21.52391.1931
@masahi
masahi
merged commit 19717aa
into
apache:mainJan 19, 2022
yuanfz98 pushed a commit
to yuanfz98/tvm
that referenced
this pull request
Jan 24, 2022
Tristan Konolige@yuanfz98
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
ed65270
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
crazydemo pushed a commit
to crazydemo/tvm
that referenced
this pull request
Jan 27, 2022
Tristan KonoligeIvy
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
21351bb
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
ylc pushed a commit
to ylc/tvm
that referenced
this pull request
Feb 16, 2022
Tristan Konolige@ylc
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
b08f596
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@masahimasahimasahi approved these changes
@mbrookhartmbrookhartmbrookhart approved these changes
@HuyuweiHuyuweiAwaiting requested review from Huyuwei
@jcf94jcf94Awaiting requested review from jcf94
@jwfrommjwfrommAwaiting requested review from jwfromm
@kevinthesunkevinthesunAwaiting requested review from kevinthesun
@LaurawlyLaurawlyAwaiting requested review from Laurawly
@vinx13vinx13Awaiting requested review from vinx13
@yzhliuyzhliuAwaiting requested review from yzhliu
@ZihengJiangZihengJiangAwaiting requested review from ZihengJiang

+1 more reviewer

@michalpiszczekmichalpiszczekmichalpiszczek approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@tkonolige@masahi@mbrookhart@michalpiszczek

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
[TOPI,x86] Improve performance on int8 conv2d on x86 - #9966
Merged
masahi merged 1 commit into
apache:mainfrom 
tkonolige:faster_int8_conv
Jan 19, 2022
Merged
[TOPI,x86] Improve performance on int8 conv2d on x86#9966
masahi merged 1 commit into
apache:mainfrom 
tkonolige:faster_int8_conv
Conversation
@tkonolige
@tkonoligetkonolige
commented
Jan 18, 2022•

edited

Loading

Copy link


Copy Markdown

Contributor
Appended fused operations in cov2d for int8 were computed in a separate loop from the main conv2d computation:
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for j in ..
out = out + fused subsequent ops

This patch moves the fused ops one more loop nesting inwards to get
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops

On quantized mobilenetv2, this results in approximately a 30% speedup.
@masahi@mbrookhart
Tristan Konolige
[TOPI,x86] Improve performance on int8 conv2d on x86
858b30c
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
masahi
masahi
approved these changes
Jan 18, 2022
View reviewed changes
@masahimasahi
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense, and good speed up!
Curious how much it helps for other models. At first I thought this would be a bigger win for larger workloads (with corresponding larger write cache). It is easy to test quantized resnet50, inception v3, or mobilenet v3 via PyTorch.
mbrookhart
mbrookhart
approved these changes
Jan 18, 2022
View reviewed changes
@mbrookhartmbrookhart
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay!
@tkonolige
tkonolige
commented
Jan 18, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
ModelBefore (ms)After (ms)
resnet505.52824.7501
inception_v33.43203.2994
mobilenetv21.52391.1931
@masahi
masahi
merged commit 19717aa
into
apache:mainJan 19, 2022
yuanfz98 pushed a commit
to yuanfz98/tvm
that referenced
this pull request
Jan 24, 2022
Tristan Konolige@yuanfz98
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
ed65270
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
crazydemo pushed a commit
to crazydemo/tvm
that referenced
this pull request
Jan 27, 2022
Tristan KonoligeIvy
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
21351bb
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
ylc pushed a commit
to ylc/tvm
that referenced
this pull request
Feb 16, 2022
Tristan Konolige@ylc
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
b08f596
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@masahimasahimasahi approved these changes
@mbrookhartmbrookhartmbrookhart approved these changes
@HuyuweiHuyuweiAwaiting requested review from Huyuwei
@jcf94jcf94Awaiting requested review from jcf94
@jwfrommjwfrommAwaiting requested review from jwfromm
@kevinthesunkevinthesunAwaiting requested review from kevinthesun
@LaurawlyLaurawlyAwaiting requested review from Laurawly
@vinx13vinx13Awaiting requested review from vinx13
@yzhliuyzhliuAwaiting requested review from yzhliu
@ZihengJiangZihengJiangAwaiting requested review from ZihengJiang

+1 more reviewer

@michalpiszczekmichalpiszczekmichalpiszczek approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@tkonolige@masahi@mbrookhart@michalpiszczek

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
[TOPI,x86] Improve performance on int8 conv2d on x86 - #9966
Merged
masahi merged 1 commit into
apache:mainfrom 
tkonolige:faster_int8_conv
Jan 19, 2022
Merged
[TOPI,x86] Improve performance on int8 conv2d on x86#9966
masahi merged 1 commit into
apache:mainfrom 
tkonolige:faster_int8_conv
Conversation
@tkonolige
@tkonoligetkonolige
commented
Jan 18, 2022•

edited

Loading

Copy link


Copy Markdown

Contributor
Appended fused operations in cov2d for int8 were computed in a separate loop from the main conv2d computation:
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for j in ..
out = out + fused subsequent ops

This patch moves the fused ops one more loop nesting inwards to get
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops

On quantized mobilenetv2, this results in approximately a 30% speedup.
@masahi@mbrookhart
Tristan Konolige
[TOPI,x86] Improve performance on int8 conv2d on x86
858b30c
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
masahi
masahi
approved these changes
Jan 18, 2022
View reviewed changes
@masahimasahi
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense, and good speed up!
Curious how much it helps for other models. At first I thought this would be a bigger win for larger workloads (with corresponding larger write cache). It is easy to test quantized resnet50, inception v3, or mobilenet v3 via PyTorch.
mbrookhart
mbrookhart
approved these changes
Jan 18, 2022
View reviewed changes
@mbrookhartmbrookhart
left a comment


Copy link


Copy Markdown

Contributor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay!
@tkonolige
tkonolige
commented
Jan 18, 2022•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
ModelBefore (ms)After (ms)
resnet505.52824.7501
inception_v33.43203.2994
mobilenetv21.52391.1931
@masahi
masahi
merged commit 19717aa
into
apache:mainJan 19, 2022
yuanfz98 pushed a commit
to yuanfz98/tvm
that referenced
this pull request
Jan 24, 2022
Tristan Konolige@yuanfz98
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
ed65270
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
crazydemo pushed a commit
to crazydemo/tvm
that referenced
this pull request
Jan 27, 2022
Tristan KonoligeIvy
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
21351bb
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
ylc pushed a commit
to ylc/tvm
that referenced
this pull request
Feb 16, 2022
Tristan Konolige@ylc
[TOPI,x86] Improve performance on int8 conv2d on x86 (apache#9966)
b08f596
Appended fused operations in cov2d for int8 were computed in a separate
loop from the main conv2d computation:
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator
for k in ..
out = out + fused subsequent ops
```
This patch moves the fused ops one more loop nesting inwards to get
```
for i in ... parallel
for j in ...
accumulator = 0
for k in ..
vectorized_multiply_add(accumulator, data, kernel)
out = accumulator + fused subsequent ops
```
On quantized mobilenetv2, this results in approximately a 30% speedup.
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@masahimasahimasahi approved these changes
@mbrookhartmbrookhartmbrookhart approved these changes
@HuyuweiHuyuweiAwaiting requested review from Huyuwei
@jcf94jcf94Awaiting requested review from jcf94
@jwfrommjwfrommAwaiting requested review from jwfromm
@kevinthesunkevinthesunAwaiting requested review from kevinthesun
@LaurawlyLaurawlyAwaiting requested review from Laurawly
@vinx13vinx13Awaiting requested review from vinx13
@yzhliuyzhliuAwaiting requested review from yzhliu
@ZihengJiangZihengJiangAwaiting requested review from ZihengJiang

+1 more reviewer

@michalpiszczekmichalpiszczekmichalpiszczek approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees


No one assigned


Labels


None yet


Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

4 participants

@tkonolige@masahi@mbrookhart@michalpiszczek