Bump PyTorch nightly pin past April 22, 2025 - #10362

Merged
swolchok merged 5 commits into
mainfrom
gh/swolchok/422/head
May 27, 2025
Merged

Bump PyTorch nightly pin past April 22, 2025#10362
swolchok merged 5 commits into
mainfrom
gh/swolchok/422/head

Conversation

@swolchok

@swolchokswolchok commented Apr 22, 2025

Copy link
Copy Markdown
Contributor

@swolchok
Update
298df23
[ghstack-poisoned]
@swolchok

swolchok commented Apr 22, 2025•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Stack from ghstack (oldest at bottom):

@pytorch-bot

pytorch-botBot commented Apr 22, 2025•
edited
Loading

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10362

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 3 Unrelated Failures

As of commit b8f6b50 with merge base 3066463 (image):

NEW FAILURES - The following jobs have failed:

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

swolchok added a commit that referenced this pull request Apr 22, 2025
@swolchok
Bump PyTorch nightly pin past April 22 2025
eef7ac1
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: 7ef4d5c
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@facebook-github-botfacebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Apr 22, 2025
@swolchok

swolchok commented Apr 22, 2025•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

"Build documentation / check-urls" failure looks spurious; will rerun when the option becomes available.
ditto "trunk / test-models-linux-aarch64 (qwen2_5, portable, linux.arm64.2xlarge) / linux-job (push)"

@cccclai

cccclai commented Apr 22, 2025

Copy link
Copy Markdown
Contributor

Can we hold on bumping? The PR pytorch/pytorch#151436 merged last week causing 50+ tests failing...

@swolchok

swolchok commented Apr 23, 2025

Copy link
Copy Markdown
ContributorAuthor

hold on bumping?

Can we hold on bumping? The PR pytorch/pytorch#151436 merged last week causing 50+ tests failing...

@cccclai OK please let me know when we can do this

@cccclai

cccclai commented Apr 23, 2025

Copy link
Copy Markdown
Contributor

@tugsbayasgalan is working on resolving the failure.

@cccclaicccclai mentioned this pull request Apr 23, 2025
Merged
@swolchok

swolchok commented Apr 25, 2025

Copy link
Copy Markdown
ContributorAuthor

ping @cccclai@tugsbayasgalan

@cccclai

cccclai commented May 13, 2025

Copy link
Copy Markdown
Contributor

ping @cccclai@tugsbayasgalan

See discussion in #10319 (comment), hopefully it's resolved soon

@cccclai

cccclai commented May 15, 2025

Copy link
Copy Markdown
Contributor

#10769 for the fix.

@swolchok
Update
d23fef6
[ghstack-poisoned]
swolchok added a commit that referenced this pull request May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
322c879
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: fbc5600
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok changed the title Bump PyTorch nightly pin past April 22 2025Bump PyTorch nightly pin past May 21, 2025May 21, 2025
@swolchok

swolchok commented May 21, 2025

Copy link
Copy Markdown
ContributorAuthor

looks like we have more breakage if I try to roll forward to the present day. let's go back to at least catching up to April 22

@swolchok
Update
a212e81
[ghstack-poisoned]
@swolchokswolchok changed the title Bump PyTorch nightly pin past May 21, 2025Bump PyTorch nightly pin past April 22, 2025May 21, 2025
swolchok added a commit that referenced this pull request May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
08dca3d
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: ca64709
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok changed the title Bump PyTorch nightly pin past April 22, 2025Bump PyTorch nightly pin past May 21, 2025May 21, 2025
@cccclai

cccclai commented May 21, 2025

Copy link
Copy Markdown
Contributor

#10769 is merged as fyi

@swolchokswolchok changed the title Bump PyTorch nightly pin past May 21, 2025Bump PyTorch nightly pin past April 22, 2025May 21, 2025
@swolchok
sync c10
089917c
[ghstack-poisoned]
swolchok added a commit that referenced this pull request May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
660b1bc
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: f50a6df
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok added the release notes: none Do not include this in the release notes label May 21, 2025
@swolchok

swolchok commented May 22, 2025

Copy link
Copy Markdown
ContributorAuthor

test-llava-runner-linux seems to be consistently failing with SIGILL :(

@swolchok

swolchok commented May 23, 2025

Copy link
Copy Markdown
ContributorAuthor

test-llava-runner-linux seems to be consistently failing with SIGILL :(

Hm, I successfully ran python -m unittest examples.models.llava.test.test_llava with this PR on a Linux box, and Google says that the runners support AVX512, so it's not something simple like that. Trying one more time.

@swolchok

swolchok commented May 23, 2025

Copy link
Copy Markdown
ContributorAuthor

if I'm reading correctly, the illegal instruction is a vgf2p8affineqb somewhere in portable_lib. this is an AVX512GFNI instruction which requires at least Ice Lake and the processor in question appears to be Cascade Lake.

@swolchok

swolchok commented May 23, 2025

Copy link
Copy Markdown
ContributorAuthor

This instruction does seem to be generated in xnn_f32_vrcopysignc_ukernel__sse2_u16 in my local build as well, I just have a machine handy that can execute it. I guess something in the pin bump must've caused us to start delegating this op to XNNPACK? CC @digantdesai

@swolchok

swolchok commented May 23, 2025•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

in xnn_f32_vrcopysignc_ukernel__sse2_u16

the symbol name here may or may not be accurate; I just found that XNNPACK has an avx512vnnigfni configuration

@swolchok

swolchok commented May 23, 2025

Copy link
Copy Markdown
ContributorAuthor

we are in fact building at least some avx512vnnigfni files per https://github.com/pytorch/executorch/actions/runs/15171458271/job/42808833402?pr=10362

however, this is not new; https://github.com/pytorch/executorch/actions/runs/15218616519/job/42809734929 (a recent trunk run from HUD) also builds them.

either 1) we are delegating some op that we weren't delegating before AND XNNPACK's gating for these GFNI kernels was always broken, 2) bumping the pytorch pin somehow magically caused ExecuTorch's copy of XNNPACK to change, or 3) something else I haven't thought of yet.

@digantdesai

digantdesai commented May 24, 2025•
edited
Loading

Copy link
Copy Markdown
Contributor

This instruction is likely coming from the QB4W gemm kernel. This kernel is only reachable through ET and not PT. Not sure why it is getting activated on a Cascade Lake CPU on the CI C5, esp on a pin bump. At compile time this kernel compilation is gated by XNNPACK_ENABLE_AVX512VNNIGFNI and at runtime it is selected if cpuinfo_has_x86_avx512vnni() && cpuinfo_has_x86_gfni().

For now you may try disabling these kernels at compile time, to unblock yourself from updating the pin. I can try to take a closer look next week. Like you, the x86 box I have at hand is also Icelake which supports these.

@swolchok
rebase (undo apparently-accidental pin bump only in install_requireme…
b8f6b50
…nts), disable XNNPACK_ENABLE_AVX512VNNIGFNI
[ghstack-poisoned]
swolchok added a commit that referenced this pull request May 27, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
6331dee
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: 6c5d261
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchok

swolchok commented May 27, 2025

Copy link
Copy Markdown
ContributorAuthor

failures seem to pattern-match to known issues on main. merging.

@swolchok
swolchok merged commit 4cf5c06 into mainMay 27, 2025
@swolchok
swolchok deleted the gh/swolchok/422/head branch May 27, 2025 17:59
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mergennachinmergennachinmergennachin approved these changes
@jackzhxngjackzhxngAwaiting requested review from jackzhxng
@GregoryComerGregoryComerAwaiting requested review from GregoryComer
@JacobSzwejbkaJacobSzwejbkaAwaiting requested review from JacobSzwejbkaJacobSzwejbka is a code owner
@lucylqlucylqAwaiting requested review from lucylq
@digantdesaidigantdesaiAwaiting requested review from digantdesaidigantdesai is a code owner
@mcr229mcr229Awaiting requested review from mcr229

Assignees

No one assigned

Labels

ciflow/trunkCLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.release notes: noneDo not include this in the release notestopic: not user facing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@swolchok@cccclai@digantdesai@mergennachin@facebook-github-bot
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Bump PyTorch nightly pin past April 22, 2025 - #10362
Merged
swolchok merged 5 commits into
mainfrom 
gh/swolchok/422/head
May 27, 2025
Merged
Bump PyTorch nightly pin past April 22, 2025#10362
swolchok merged 5 commits into
mainfrom 
gh/swolchok/422/head
Conversation
@swolchok
@swolchokswolchok
commented
Apr 22, 2025

Copy link


Copy Markdown

Contributor
@swolchok
Update
298df23
[ghstack-poisoned]
@swolchok
swolchok
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Stack from ghstack (oldest at bottom):
@pytorch-bot
pytorch-botBot
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10362
Note: Links to docs will display an error until the docs builds have been completed.
❌ 2 New Failures, 3 Unrelated Failures
As of commit b8f6b50 with merge base 3066463 (image):
NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following jobs failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
swolchok added a commit
that referenced
this pull request
Apr 22, 2025
@swolchok
Bump PyTorch nightly pin past April 22 2025
eef7ac1
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: 7ef4d5c
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@facebook-github-botfacebook-github-bot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Apr 22, 2025
@swolchok
swolchok
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
"Build documentation / check-urls" failure looks spurious; will rerun when the option becomes available.

ditto "trunk / test-models-linux-aarch64 (qwen2_5, portable, linux.arm64.2xlarge) / linux-job (push)"
@cccclai
cccclai
commented
Apr 22, 2025

Copy link


Copy Markdown

Contributor
Can we hold on bumping? The PR pytorch/pytorch#151436 merged last week causing 50+ tests failing...
@swolchok
swolchok
commented
Apr 23, 2025

Copy link


Copy Markdown

ContributorAuthor
hold on bumping?
Can we hold on bumping? The PR pytorch/pytorch#151436 merged last week causing 50+ tests failing...
@cccclai OK please let me know when we can do this
@cccclai
cccclai
commented
Apr 23, 2025

Copy link


Copy Markdown

Contributor
@tugsbayasgalan is working on resolving the failure.
@cccclaicccclai
mentioned this pull request
Apr 23, 2025
 Merged

@swolchok
swolchok
commented
Apr 25, 2025

Copy link


Copy Markdown

ContributorAuthor
ping @cccclai@tugsbayasgalan
@cccclai
cccclai
commented
May 13, 2025

Copy link


Copy Markdown

Contributor
ping @cccclai@tugsbayasgalan
See discussion in #10319 (comment), hopefully it's resolved soon
@cccclai
cccclai
commented
May 15, 2025

Copy link


Copy Markdown

Contributor
#10769 for the fix.
@swolchok
Update
d23fef6
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
322c879
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: fbc5600
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past April 22 2025Bump PyTorch nightly pin past May 21, 2025May 21, 2025
@swolchok
swolchok
commented
May 21, 2025

Copy link


Copy Markdown

ContributorAuthor
looks like we have more breakage if I try to roll forward to the present day. let's go back to at least catching up to April 22
@swolchok
Update
a212e81
[ghstack-poisoned]
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past May 21, 2025Bump PyTorch nightly pin past April 22, 2025May 21, 2025
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
08dca3d
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: ca64709
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past April 22, 2025Bump PyTorch nightly pin past May 21, 2025May 21, 2025
@cccclai
cccclai
commented
May 21, 2025

Copy link


Copy Markdown

Contributor
#10769 is merged as fyi
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past May 21, 2025Bump PyTorch nightly pin past April 22, 2025May 21, 2025
@swolchok
sync c10
089917c
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
660b1bc
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: f50a6df
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
added
the 
release notes: none
Do not include this in the release notes
label
May 21, 2025
@swolchok
swolchok
commented
May 22, 2025

Copy link


Copy Markdown

ContributorAuthor
test-llava-runner-linux seems to be consistently failing with SIGILL :(
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
test-llava-runner-linux seems to be consistently failing with SIGILL :(
Hm, I successfully ran python -m unittest examples.models.llava.test.test_llava with this PR on a Linux box, and Google says that the runners support AVX512, so it's not something simple like that. Trying one more time.
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
if I'm reading correctly, the illegal instruction is a vgf2p8affineqb somewhere in portable_lib. this is an AVX512GFNI instruction which requires at least Ice Lake and the processor in question appears to be Cascade Lake.
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
This instruction does seem to be generated in xnn_f32_vrcopysignc_ukernel__sse2_u16 in my local build as well, I just have a machine handy that can execute it. I guess something in the pin bump must've caused us to start delegating this op to XNNPACK? CC @digantdesai
@swolchok
swolchok
commented
May 23, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
in xnn_f32_vrcopysignc_ukernel__sse2_u16
the symbol name here may or may not be accurate; I just found that XNNPACK has an avx512vnnigfni configuration
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
we are in fact building at least some avx512vnnigfni files per https://github.com/pytorch/executorch/actions/runs/15171458271/job/42808833402?pr=10362
however, this is not new; https://github.com/pytorch/executorch/actions/runs/15218616519/job/42809734929 (a recent trunk run from HUD) also builds them.
either 1) we are delegating some op that we weren't delegating before AND XNNPACK's gating for these GFNI kernels was always broken, 2) bumping the pytorch pin somehow magically caused ExecuTorch's copy of XNNPACK to change, or 3) something else I haven't thought of yet.
@digantdesai
digantdesai
commented
May 24, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
This instruction is likely coming from the QB4W gemm kernel. This kernel is only reachable through ET and not PT. Not sure why it is getting activated on a Cascade Lake CPU on the CI C5, esp on a pin bump. At compile time this kernel compilation is gated by XNNPACK_ENABLE_AVX512VNNIGFNI and at runtime it is selected if cpuinfo_has_x86_avx512vnni() && cpuinfo_has_x86_gfni().
For now you may try disabling these kernels at compile time, to unblock yourself from updating the pin. I can try to take a closer look next week. Like you, the x86 box I have at hand is also Icelake which supports these.
@swolchok
rebase (undo apparently-accidental pin bump only in install_requireme…
b8f6b50
…nts), disable XNNPACK_ENABLE_AVX512VNNIGFNI
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 27, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
6331dee
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: 6c5d261
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchok
swolchok
commented
May 27, 2025

Copy link


Copy Markdown

ContributorAuthor
failures seem to pattern-match to known issues on main. merging.
@swolchok
swolchok
merged commit 4cf5c06
into
mainMay 27, 2025
@swolchok
swolchok
deleted the

gh/swolchok/422/head

branch
May 27, 2025 17:59
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mergennachinmergennachinmergennachin approved these changes
@jackzhxngjackzhxngAwaiting requested review from jackzhxng
@GregoryComerGregoryComerAwaiting requested review from GregoryComer
@JacobSzwejbkaJacobSzwejbkaAwaiting requested review from JacobSzwejbkaJacobSzwejbka is a code owner
@lucylqlucylqAwaiting requested review from lucylq
@digantdesaidigantdesaiAwaiting requested review from digantdesaidigantdesai is a code owner
@mcr229mcr229Awaiting requested review from mcr229

Assignees


No one assigned


Labels

ciflow/trunkCLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.release notes: noneDo not include this in the release notestopic: not user facing

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@swolchok@cccclai@digantdesai@mergennachin@facebook-github-bot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Bump PyTorch nightly pin past April 22, 2025 - #10362
Merged
swolchok merged 5 commits into
mainfrom 
gh/swolchok/422/head
May 27, 2025
Merged
Bump PyTorch nightly pin past April 22, 2025#10362
swolchok merged 5 commits into
mainfrom 
gh/swolchok/422/head
Conversation
@swolchok
@swolchokswolchok
commented
Apr 22, 2025

Copy link


Copy Markdown

Contributor
@swolchok
Update
298df23
[ghstack-poisoned]
@swolchok
swolchok
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Stack from ghstack (oldest at bottom):
@pytorch-bot
pytorch-botBot
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10362
Note: Links to docs will display an error until the docs builds have been completed.
❌ 2 New Failures, 3 Unrelated Failures
As of commit b8f6b50 with merge base 3066463 (image):
NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following jobs failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
swolchok added a commit
that referenced
this pull request
Apr 22, 2025
@swolchok
Bump PyTorch nightly pin past April 22 2025
eef7ac1
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: 7ef4d5c
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@facebook-github-botfacebook-github-bot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Apr 22, 2025
@swolchok
swolchok
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
"Build documentation / check-urls" failure looks spurious; will rerun when the option becomes available.

ditto "trunk / test-models-linux-aarch64 (qwen2_5, portable, linux.arm64.2xlarge) / linux-job (push)"
@cccclai
cccclai
commented
Apr 22, 2025

Copy link


Copy Markdown

Contributor
Can we hold on bumping? The PR pytorch/pytorch#151436 merged last week causing 50+ tests failing...
@swolchok
swolchok
commented
Apr 23, 2025

Copy link


Copy Markdown

ContributorAuthor
hold on bumping?
Can we hold on bumping? The PR pytorch/pytorch#151436 merged last week causing 50+ tests failing...
@cccclai OK please let me know when we can do this
@cccclai
cccclai
commented
Apr 23, 2025

Copy link


Copy Markdown

Contributor
@tugsbayasgalan is working on resolving the failure.
@cccclaicccclai
mentioned this pull request
Apr 23, 2025
 Merged

@swolchok
swolchok
commented
Apr 25, 2025

Copy link


Copy Markdown

ContributorAuthor
ping @cccclai@tugsbayasgalan
@cccclai
cccclai
commented
May 13, 2025

Copy link


Copy Markdown

Contributor
ping @cccclai@tugsbayasgalan
See discussion in #10319 (comment), hopefully it's resolved soon
@cccclai
cccclai
commented
May 15, 2025

Copy link


Copy Markdown

Contributor
#10769 for the fix.
@swolchok
Update
d23fef6
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
322c879
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: fbc5600
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past April 22 2025Bump PyTorch nightly pin past May 21, 2025May 21, 2025
@swolchok
swolchok
commented
May 21, 2025

Copy link


Copy Markdown

ContributorAuthor
looks like we have more breakage if I try to roll forward to the present day. let's go back to at least catching up to April 22
@swolchok
Update
a212e81
[ghstack-poisoned]
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past May 21, 2025Bump PyTorch nightly pin past April 22, 2025May 21, 2025
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
08dca3d
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: ca64709
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past April 22, 2025Bump PyTorch nightly pin past May 21, 2025May 21, 2025
@cccclai
cccclai
commented
May 21, 2025

Copy link


Copy Markdown

Contributor
#10769 is merged as fyi
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past May 21, 2025Bump PyTorch nightly pin past April 22, 2025May 21, 2025
@swolchok
sync c10
089917c
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
660b1bc
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: f50a6df
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
added
the 
release notes: none
Do not include this in the release notes
label
May 21, 2025
@swolchok
swolchok
commented
May 22, 2025

Copy link


Copy Markdown

ContributorAuthor
test-llava-runner-linux seems to be consistently failing with SIGILL :(
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
test-llava-runner-linux seems to be consistently failing with SIGILL :(
Hm, I successfully ran python -m unittest examples.models.llava.test.test_llava with this PR on a Linux box, and Google says that the runners support AVX512, so it's not something simple like that. Trying one more time.
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
if I'm reading correctly, the illegal instruction is a vgf2p8affineqb somewhere in portable_lib. this is an AVX512GFNI instruction which requires at least Ice Lake and the processor in question appears to be Cascade Lake.
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
This instruction does seem to be generated in xnn_f32_vrcopysignc_ukernel__sse2_u16 in my local build as well, I just have a machine handy that can execute it. I guess something in the pin bump must've caused us to start delegating this op to XNNPACK? CC @digantdesai
@swolchok
swolchok
commented
May 23, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
in xnn_f32_vrcopysignc_ukernel__sse2_u16
the symbol name here may or may not be accurate; I just found that XNNPACK has an avx512vnnigfni configuration
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
we are in fact building at least some avx512vnnigfni files per https://github.com/pytorch/executorch/actions/runs/15171458271/job/42808833402?pr=10362
however, this is not new; https://github.com/pytorch/executorch/actions/runs/15218616519/job/42809734929 (a recent trunk run from HUD) also builds them.
either 1) we are delegating some op that we weren't delegating before AND XNNPACK's gating for these GFNI kernels was always broken, 2) bumping the pytorch pin somehow magically caused ExecuTorch's copy of XNNPACK to change, or 3) something else I haven't thought of yet.
@digantdesai
digantdesai
commented
May 24, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
This instruction is likely coming from the QB4W gemm kernel. This kernel is only reachable through ET and not PT. Not sure why it is getting activated on a Cascade Lake CPU on the CI C5, esp on a pin bump. At compile time this kernel compilation is gated by XNNPACK_ENABLE_AVX512VNNIGFNI and at runtime it is selected if cpuinfo_has_x86_avx512vnni() && cpuinfo_has_x86_gfni().
For now you may try disabling these kernels at compile time, to unblock yourself from updating the pin. I can try to take a closer look next week. Like you, the x86 box I have at hand is also Icelake which supports these.
@swolchok
rebase (undo apparently-accidental pin bump only in install_requireme…
b8f6b50
…nts), disable XNNPACK_ENABLE_AVX512VNNIGFNI
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 27, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
6331dee
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: 6c5d261
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchok
swolchok
commented
May 27, 2025

Copy link


Copy Markdown

ContributorAuthor
failures seem to pattern-match to known issues on main. merging.
@swolchok
swolchok
merged commit 4cf5c06
into
mainMay 27, 2025
@swolchok
swolchok
deleted the

gh/swolchok/422/head

branch
May 27, 2025 17:59
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mergennachinmergennachinmergennachin approved these changes
@jackzhxngjackzhxngAwaiting requested review from jackzhxng
@GregoryComerGregoryComerAwaiting requested review from GregoryComer
@JacobSzwejbkaJacobSzwejbkaAwaiting requested review from JacobSzwejbkaJacobSzwejbka is a code owner
@lucylqlucylqAwaiting requested review from lucylq
@digantdesaidigantdesaiAwaiting requested review from digantdesaidigantdesai is a code owner
@mcr229mcr229Awaiting requested review from mcr229

Assignees


No one assigned


Labels

ciflow/trunkCLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.release notes: noneDo not include this in the release notestopic: not user facing

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@swolchok@cccclai@digantdesai@mergennachin@facebook-github-bot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Bump PyTorch nightly pin past April 22, 2025 - #10362
Merged
swolchok merged 5 commits into
mainfrom 
gh/swolchok/422/head
May 27, 2025
Merged
Bump PyTorch nightly pin past April 22, 2025#10362
swolchok merged 5 commits into
mainfrom 
gh/swolchok/422/head
Conversation
@swolchok
@swolchokswolchok
commented
Apr 22, 2025

Copy link


Copy Markdown

Contributor
@swolchok
Update
298df23
[ghstack-poisoned]
@swolchok
swolchok
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Stack from ghstack (oldest at bottom):
@pytorch-bot
pytorch-botBot
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10362
Note: Links to docs will display an error until the docs builds have been completed.
❌ 2 New Failures, 3 Unrelated Failures
As of commit b8f6b50 with merge base 3066463 (image):
NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following jobs failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
swolchok added a commit
that referenced
this pull request
Apr 22, 2025
@swolchok
Bump PyTorch nightly pin past April 22 2025
eef7ac1
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: 7ef4d5c
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@facebook-github-botfacebook-github-bot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Apr 22, 2025
@swolchok
swolchok
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
"Build documentation / check-urls" failure looks spurious; will rerun when the option becomes available.

ditto "trunk / test-models-linux-aarch64 (qwen2_5, portable, linux.arm64.2xlarge) / linux-job (push)"
@cccclai
cccclai
commented
Apr 22, 2025

Copy link


Copy Markdown

Contributor
Can we hold on bumping? The PR pytorch/pytorch#151436 merged last week causing 50+ tests failing...
@swolchok
swolchok
commented
Apr 23, 2025

Copy link


Copy Markdown

ContributorAuthor
hold on bumping?
Can we hold on bumping? The PR pytorch/pytorch#151436 merged last week causing 50+ tests failing...
@cccclai OK please let me know when we can do this
@cccclai
cccclai
commented
Apr 23, 2025

Copy link


Copy Markdown

Contributor
@tugsbayasgalan is working on resolving the failure.
@cccclaicccclai
mentioned this pull request
Apr 23, 2025
 Merged

@swolchok
swolchok
commented
Apr 25, 2025

Copy link


Copy Markdown

ContributorAuthor
ping @cccclai@tugsbayasgalan
@cccclai
cccclai
commented
May 13, 2025

Copy link


Copy Markdown

Contributor
ping @cccclai@tugsbayasgalan
See discussion in #10319 (comment), hopefully it's resolved soon
@cccclai
cccclai
commented
May 15, 2025

Copy link


Copy Markdown

Contributor
#10769 for the fix.
@swolchok
Update
d23fef6
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
322c879
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: fbc5600
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past April 22 2025Bump PyTorch nightly pin past May 21, 2025May 21, 2025
@swolchok
swolchok
commented
May 21, 2025

Copy link


Copy Markdown

ContributorAuthor
looks like we have more breakage if I try to roll forward to the present day. let's go back to at least catching up to April 22
@swolchok
Update
a212e81
[ghstack-poisoned]
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past May 21, 2025Bump PyTorch nightly pin past April 22, 2025May 21, 2025
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
08dca3d
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: ca64709
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past April 22, 2025Bump PyTorch nightly pin past May 21, 2025May 21, 2025
@cccclai
cccclai
commented
May 21, 2025

Copy link


Copy Markdown

Contributor
#10769 is merged as fyi
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past May 21, 2025Bump PyTorch nightly pin past April 22, 2025May 21, 2025
@swolchok
sync c10
089917c
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
660b1bc
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: f50a6df
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
added
the 
release notes: none
Do not include this in the release notes
label
May 21, 2025
@swolchok
swolchok
commented
May 22, 2025

Copy link


Copy Markdown

ContributorAuthor
test-llava-runner-linux seems to be consistently failing with SIGILL :(
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
test-llava-runner-linux seems to be consistently failing with SIGILL :(
Hm, I successfully ran python -m unittest examples.models.llava.test.test_llava with this PR on a Linux box, and Google says that the runners support AVX512, so it's not something simple like that. Trying one more time.
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
if I'm reading correctly, the illegal instruction is a vgf2p8affineqb somewhere in portable_lib. this is an AVX512GFNI instruction which requires at least Ice Lake and the processor in question appears to be Cascade Lake.
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
This instruction does seem to be generated in xnn_f32_vrcopysignc_ukernel__sse2_u16 in my local build as well, I just have a machine handy that can execute it. I guess something in the pin bump must've caused us to start delegating this op to XNNPACK? CC @digantdesai
@swolchok
swolchok
commented
May 23, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
in xnn_f32_vrcopysignc_ukernel__sse2_u16
the symbol name here may or may not be accurate; I just found that XNNPACK has an avx512vnnigfni configuration
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
we are in fact building at least some avx512vnnigfni files per https://github.com/pytorch/executorch/actions/runs/15171458271/job/42808833402?pr=10362
however, this is not new; https://github.com/pytorch/executorch/actions/runs/15218616519/job/42809734929 (a recent trunk run from HUD) also builds them.
either 1) we are delegating some op that we weren't delegating before AND XNNPACK's gating for these GFNI kernels was always broken, 2) bumping the pytorch pin somehow magically caused ExecuTorch's copy of XNNPACK to change, or 3) something else I haven't thought of yet.
@digantdesai
digantdesai
commented
May 24, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
This instruction is likely coming from the QB4W gemm kernel. This kernel is only reachable through ET and not PT. Not sure why it is getting activated on a Cascade Lake CPU on the CI C5, esp on a pin bump. At compile time this kernel compilation is gated by XNNPACK_ENABLE_AVX512VNNIGFNI and at runtime it is selected if cpuinfo_has_x86_avx512vnni() && cpuinfo_has_x86_gfni().
For now you may try disabling these kernels at compile time, to unblock yourself from updating the pin. I can try to take a closer look next week. Like you, the x86 box I have at hand is also Icelake which supports these.
@swolchok
rebase (undo apparently-accidental pin bump only in install_requireme…
b8f6b50
…nts), disable XNNPACK_ENABLE_AVX512VNNIGFNI
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 27, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
6331dee
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: 6c5d261
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchok
swolchok
commented
May 27, 2025

Copy link


Copy Markdown

ContributorAuthor
failures seem to pattern-match to known issues on main. merging.
@swolchok
swolchok
merged commit 4cf5c06
into
mainMay 27, 2025
@swolchok
swolchok
deleted the

gh/swolchok/422/head

branch
May 27, 2025 17:59
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mergennachinmergennachinmergennachin approved these changes
@jackzhxngjackzhxngAwaiting requested review from jackzhxng
@GregoryComerGregoryComerAwaiting requested review from GregoryComer
@JacobSzwejbkaJacobSzwejbkaAwaiting requested review from JacobSzwejbkaJacobSzwejbka is a code owner
@lucylqlucylqAwaiting requested review from lucylq
@digantdesaidigantdesaiAwaiting requested review from digantdesaidigantdesai is a code owner
@mcr229mcr229Awaiting requested review from mcr229

Assignees


No one assigned


Labels

ciflow/trunkCLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.release notes: noneDo not include this in the release notestopic: not user facing

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@swolchok@cccclai@digantdesai@mergennachin@facebook-github-bot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Bump PyTorch nightly pin past April 22, 2025 - #10362
Merged
swolchok merged 5 commits into
mainfrom 
gh/swolchok/422/head
May 27, 2025
Merged
Bump PyTorch nightly pin past April 22, 2025#10362
swolchok merged 5 commits into
mainfrom 
gh/swolchok/422/head
Conversation
@swolchok
@swolchokswolchok
commented
Apr 22, 2025

Copy link


Copy Markdown

Contributor
@swolchok
Update
298df23
[ghstack-poisoned]
@swolchok
swolchok
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Stack from ghstack (oldest at bottom):
@pytorch-bot
pytorch-botBot
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10362
Note: Links to docs will display an error until the docs builds have been completed.
❌ 2 New Failures, 3 Unrelated Failures
As of commit b8f6b50 with merge base 3066463 (image):
NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following jobs failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
swolchok added a commit
that referenced
this pull request
Apr 22, 2025
@swolchok
Bump PyTorch nightly pin past April 22 2025
eef7ac1
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: 7ef4d5c
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@facebook-github-botfacebook-github-bot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Apr 22, 2025
@swolchok
swolchok
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
"Build documentation / check-urls" failure looks spurious; will rerun when the option becomes available.

ditto "trunk / test-models-linux-aarch64 (qwen2_5, portable, linux.arm64.2xlarge) / linux-job (push)"
@cccclai
cccclai
commented
Apr 22, 2025

Copy link


Copy Markdown

Contributor
Can we hold on bumping? The PR pytorch/pytorch#151436 merged last week causing 50+ tests failing...
@swolchok
swolchok
commented
Apr 23, 2025

Copy link


Copy Markdown

ContributorAuthor
hold on bumping?
Can we hold on bumping? The PR pytorch/pytorch#151436 merged last week causing 50+ tests failing...
@cccclai OK please let me know when we can do this
@cccclai
cccclai
commented
Apr 23, 2025

Copy link


Copy Markdown

Contributor
@tugsbayasgalan is working on resolving the failure.
@cccclaicccclai
mentioned this pull request
Apr 23, 2025
 Merged

@swolchok
swolchok
commented
Apr 25, 2025

Copy link


Copy Markdown

ContributorAuthor
ping @cccclai@tugsbayasgalan
@cccclai
cccclai
commented
May 13, 2025

Copy link


Copy Markdown

Contributor
ping @cccclai@tugsbayasgalan
See discussion in #10319 (comment), hopefully it's resolved soon
@cccclai
cccclai
commented
May 15, 2025

Copy link


Copy Markdown

Contributor
#10769 for the fix.
@swolchok
Update
d23fef6
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
322c879
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: fbc5600
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past April 22 2025Bump PyTorch nightly pin past May 21, 2025May 21, 2025
@swolchok
swolchok
commented
May 21, 2025

Copy link


Copy Markdown

ContributorAuthor
looks like we have more breakage if I try to roll forward to the present day. let's go back to at least catching up to April 22
@swolchok
Update
a212e81
[ghstack-poisoned]
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past May 21, 2025Bump PyTorch nightly pin past April 22, 2025May 21, 2025
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
08dca3d
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: ca64709
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past April 22, 2025Bump PyTorch nightly pin past May 21, 2025May 21, 2025
@cccclai
cccclai
commented
May 21, 2025

Copy link


Copy Markdown

Contributor
#10769 is merged as fyi
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past May 21, 2025Bump PyTorch nightly pin past April 22, 2025May 21, 2025
@swolchok
sync c10
089917c
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
660b1bc
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: f50a6df
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
added
the 
release notes: none
Do not include this in the release notes
label
May 21, 2025
@swolchok
swolchok
commented
May 22, 2025

Copy link


Copy Markdown

ContributorAuthor
test-llava-runner-linux seems to be consistently failing with SIGILL :(
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
test-llava-runner-linux seems to be consistently failing with SIGILL :(
Hm, I successfully ran python -m unittest examples.models.llava.test.test_llava with this PR on a Linux box, and Google says that the runners support AVX512, so it's not something simple like that. Trying one more time.
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
if I'm reading correctly, the illegal instruction is a vgf2p8affineqb somewhere in portable_lib. this is an AVX512GFNI instruction which requires at least Ice Lake and the processor in question appears to be Cascade Lake.
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
This instruction does seem to be generated in xnn_f32_vrcopysignc_ukernel__sse2_u16 in my local build as well, I just have a machine handy that can execute it. I guess something in the pin bump must've caused us to start delegating this op to XNNPACK? CC @digantdesai
@swolchok
swolchok
commented
May 23, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
in xnn_f32_vrcopysignc_ukernel__sse2_u16
the symbol name here may or may not be accurate; I just found that XNNPACK has an avx512vnnigfni configuration
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
we are in fact building at least some avx512vnnigfni files per https://github.com/pytorch/executorch/actions/runs/15171458271/job/42808833402?pr=10362
however, this is not new; https://github.com/pytorch/executorch/actions/runs/15218616519/job/42809734929 (a recent trunk run from HUD) also builds them.
either 1) we are delegating some op that we weren't delegating before AND XNNPACK's gating for these GFNI kernels was always broken, 2) bumping the pytorch pin somehow magically caused ExecuTorch's copy of XNNPACK to change, or 3) something else I haven't thought of yet.
@digantdesai
digantdesai
commented
May 24, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
This instruction is likely coming from the QB4W gemm kernel. This kernel is only reachable through ET and not PT. Not sure why it is getting activated on a Cascade Lake CPU on the CI C5, esp on a pin bump. At compile time this kernel compilation is gated by XNNPACK_ENABLE_AVX512VNNIGFNI and at runtime it is selected if cpuinfo_has_x86_avx512vnni() && cpuinfo_has_x86_gfni().
For now you may try disabling these kernels at compile time, to unblock yourself from updating the pin. I can try to take a closer look next week. Like you, the x86 box I have at hand is also Icelake which supports these.
@swolchok
rebase (undo apparently-accidental pin bump only in install_requireme…
b8f6b50
…nts), disable XNNPACK_ENABLE_AVX512VNNIGFNI
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 27, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
6331dee
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: 6c5d261
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchok
swolchok
commented
May 27, 2025

Copy link


Copy Markdown

ContributorAuthor
failures seem to pattern-match to known issues on main. merging.
@swolchok
swolchok
merged commit 4cf5c06
into
mainMay 27, 2025
@swolchok
swolchok
deleted the

gh/swolchok/422/head

branch
May 27, 2025 17:59
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mergennachinmergennachinmergennachin approved these changes
@jackzhxngjackzhxngAwaiting requested review from jackzhxng
@GregoryComerGregoryComerAwaiting requested review from GregoryComer
@JacobSzwejbkaJacobSzwejbkaAwaiting requested review from JacobSzwejbkaJacobSzwejbka is a code owner
@lucylqlucylqAwaiting requested review from lucylq
@digantdesaidigantdesaiAwaiting requested review from digantdesaidigantdesai is a code owner
@mcr229mcr229Awaiting requested review from mcr229

Assignees


No one assigned


Labels

ciflow/trunkCLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.release notes: noneDo not include this in the release notestopic: not user facing

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@swolchok@cccclai@digantdesai@mergennachin@facebook-github-bot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Bump PyTorch nightly pin past April 22, 2025 - #10362
Merged
swolchok merged 5 commits into
mainfrom 
gh/swolchok/422/head
May 27, 2025
Merged
Bump PyTorch nightly pin past April 22, 2025#10362
swolchok merged 5 commits into
mainfrom 
gh/swolchok/422/head
Conversation
@swolchok
@swolchokswolchok
commented
Apr 22, 2025

Copy link


Copy Markdown

Contributor
@swolchok
Update
298df23
[ghstack-poisoned]
@swolchok
swolchok
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Stack from ghstack (oldest at bottom):
@pytorch-bot
pytorch-botBot
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10362
Note: Links to docs will display an error until the docs builds have been completed.
❌ 2 New Failures, 3 Unrelated Failures
As of commit b8f6b50 with merge base 3066463 (image):
NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following jobs failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
swolchok added a commit
that referenced
this pull request
Apr 22, 2025
@swolchok
Bump PyTorch nightly pin past April 22 2025
eef7ac1
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: 7ef4d5c
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@facebook-github-botfacebook-github-bot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Apr 22, 2025
@swolchok
swolchok
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
"Build documentation / check-urls" failure looks spurious; will rerun when the option becomes available.

ditto "trunk / test-models-linux-aarch64 (qwen2_5, portable, linux.arm64.2xlarge) / linux-job (push)"
@cccclai
cccclai
commented
Apr 22, 2025

Copy link


Copy Markdown

Contributor
Can we hold on bumping? The PR pytorch/pytorch#151436 merged last week causing 50+ tests failing...
@swolchok
swolchok
commented
Apr 23, 2025

Copy link


Copy Markdown

ContributorAuthor
hold on bumping?
Can we hold on bumping? The PR pytorch/pytorch#151436 merged last week causing 50+ tests failing...
@cccclai OK please let me know when we can do this
@cccclai
cccclai
commented
Apr 23, 2025

Copy link


Copy Markdown

Contributor
@tugsbayasgalan is working on resolving the failure.
@cccclaicccclai
mentioned this pull request
Apr 23, 2025
 Merged

@swolchok
swolchok
commented
Apr 25, 2025

Copy link


Copy Markdown

ContributorAuthor
ping @cccclai@tugsbayasgalan
@cccclai
cccclai
commented
May 13, 2025

Copy link


Copy Markdown

Contributor
ping @cccclai@tugsbayasgalan
See discussion in #10319 (comment), hopefully it's resolved soon
@cccclai
cccclai
commented
May 15, 2025

Copy link


Copy Markdown

Contributor
#10769 for the fix.
@swolchok
Update
d23fef6
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
322c879
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: fbc5600
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past April 22 2025Bump PyTorch nightly pin past May 21, 2025May 21, 2025
@swolchok
swolchok
commented
May 21, 2025

Copy link


Copy Markdown

ContributorAuthor
looks like we have more breakage if I try to roll forward to the present day. let's go back to at least catching up to April 22
@swolchok
Update
a212e81
[ghstack-poisoned]
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past May 21, 2025Bump PyTorch nightly pin past April 22, 2025May 21, 2025
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
08dca3d
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: ca64709
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past April 22, 2025Bump PyTorch nightly pin past May 21, 2025May 21, 2025
@cccclai
cccclai
commented
May 21, 2025

Copy link


Copy Markdown

Contributor
#10769 is merged as fyi
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past May 21, 2025Bump PyTorch nightly pin past April 22, 2025May 21, 2025
@swolchok
sync c10
089917c
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
660b1bc
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: f50a6df
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
added
the 
release notes: none
Do not include this in the release notes
label
May 21, 2025
@swolchok
swolchok
commented
May 22, 2025

Copy link


Copy Markdown

ContributorAuthor
test-llava-runner-linux seems to be consistently failing with SIGILL :(
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
test-llava-runner-linux seems to be consistently failing with SIGILL :(
Hm, I successfully ran python -m unittest examples.models.llava.test.test_llava with this PR on a Linux box, and Google says that the runners support AVX512, so it's not something simple like that. Trying one more time.
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
if I'm reading correctly, the illegal instruction is a vgf2p8affineqb somewhere in portable_lib. this is an AVX512GFNI instruction which requires at least Ice Lake and the processor in question appears to be Cascade Lake.
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
This instruction does seem to be generated in xnn_f32_vrcopysignc_ukernel__sse2_u16 in my local build as well, I just have a machine handy that can execute it. I guess something in the pin bump must've caused us to start delegating this op to XNNPACK? CC @digantdesai
@swolchok
swolchok
commented
May 23, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
in xnn_f32_vrcopysignc_ukernel__sse2_u16
the symbol name here may or may not be accurate; I just found that XNNPACK has an avx512vnnigfni configuration
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
we are in fact building at least some avx512vnnigfni files per https://github.com/pytorch/executorch/actions/runs/15171458271/job/42808833402?pr=10362
however, this is not new; https://github.com/pytorch/executorch/actions/runs/15218616519/job/42809734929 (a recent trunk run from HUD) also builds them.
either 1) we are delegating some op that we weren't delegating before AND XNNPACK's gating for these GFNI kernels was always broken, 2) bumping the pytorch pin somehow magically caused ExecuTorch's copy of XNNPACK to change, or 3) something else I haven't thought of yet.
@digantdesai
digantdesai
commented
May 24, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
This instruction is likely coming from the QB4W gemm kernel. This kernel is only reachable through ET and not PT. Not sure why it is getting activated on a Cascade Lake CPU on the CI C5, esp on a pin bump. At compile time this kernel compilation is gated by XNNPACK_ENABLE_AVX512VNNIGFNI and at runtime it is selected if cpuinfo_has_x86_avx512vnni() && cpuinfo_has_x86_gfni().
For now you may try disabling these kernels at compile time, to unblock yourself from updating the pin. I can try to take a closer look next week. Like you, the x86 box I have at hand is also Icelake which supports these.
@swolchok
rebase (undo apparently-accidental pin bump only in install_requireme…
b8f6b50
…nts), disable XNNPACK_ENABLE_AVX512VNNIGFNI
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 27, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
6331dee
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: 6c5d261
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchok
swolchok
commented
May 27, 2025

Copy link


Copy Markdown

ContributorAuthor
failures seem to pattern-match to known issues on main. merging.
@swolchok
swolchok
merged commit 4cf5c06
into
mainMay 27, 2025
@swolchok
swolchok
deleted the

gh/swolchok/422/head

branch
May 27, 2025 17:59
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mergennachinmergennachinmergennachin approved these changes
@jackzhxngjackzhxngAwaiting requested review from jackzhxng
@GregoryComerGregoryComerAwaiting requested review from GregoryComer
@JacobSzwejbkaJacobSzwejbkaAwaiting requested review from JacobSzwejbkaJacobSzwejbka is a code owner
@lucylqlucylqAwaiting requested review from lucylq
@digantdesaidigantdesaiAwaiting requested review from digantdesaidigantdesai is a code owner
@mcr229mcr229Awaiting requested review from mcr229

Assignees


No one assigned


Labels

ciflow/trunkCLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.release notes: noneDo not include this in the release notestopic: not user facing

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@swolchok@cccclai@digantdesai@mergennachin@facebook-github-bot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Bump PyTorch nightly pin past April 22, 2025 - #10362
Merged
swolchok merged 5 commits into
mainfrom 
gh/swolchok/422/head
May 27, 2025
Merged
Bump PyTorch nightly pin past April 22, 2025#10362
swolchok merged 5 commits into
mainfrom 
gh/swolchok/422/head
Conversation
@swolchok
@swolchokswolchok
commented
Apr 22, 2025

Copy link


Copy Markdown

Contributor
@swolchok
Update
298df23
[ghstack-poisoned]
@swolchok
swolchok
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Stack from ghstack (oldest at bottom):
@pytorch-bot
pytorch-botBot
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10362
Note: Links to docs will display an error until the docs builds have been completed.
❌ 2 New Failures, 3 Unrelated Failures
As of commit b8f6b50 with merge base 3066463 (image):
NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following jobs failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
swolchok added a commit
that referenced
this pull request
Apr 22, 2025
@swolchok
Bump PyTorch nightly pin past April 22 2025
eef7ac1
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: 7ef4d5c
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@facebook-github-botfacebook-github-bot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Apr 22, 2025
@swolchok
swolchok
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
"Build documentation / check-urls" failure looks spurious; will rerun when the option becomes available.

ditto "trunk / test-models-linux-aarch64 (qwen2_5, portable, linux.arm64.2xlarge) / linux-job (push)"
@cccclai
cccclai
commented
Apr 22, 2025

Copy link


Copy Markdown

Contributor
Can we hold on bumping? The PR pytorch/pytorch#151436 merged last week causing 50+ tests failing...
@swolchok
swolchok
commented
Apr 23, 2025

Copy link


Copy Markdown

ContributorAuthor
hold on bumping?
Can we hold on bumping? The PR pytorch/pytorch#151436 merged last week causing 50+ tests failing...
@cccclai OK please let me know when we can do this
@cccclai
cccclai
commented
Apr 23, 2025

Copy link


Copy Markdown

Contributor
@tugsbayasgalan is working on resolving the failure.
@cccclaicccclai
mentioned this pull request
Apr 23, 2025
 Merged

@swolchok
swolchok
commented
Apr 25, 2025

Copy link


Copy Markdown

ContributorAuthor
ping @cccclai@tugsbayasgalan
@cccclai
cccclai
commented
May 13, 2025

Copy link


Copy Markdown

Contributor
ping @cccclai@tugsbayasgalan
See discussion in #10319 (comment), hopefully it's resolved soon
@cccclai
cccclai
commented
May 15, 2025

Copy link


Copy Markdown

Contributor
#10769 for the fix.
@swolchok
Update
d23fef6
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
322c879
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: fbc5600
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past April 22 2025Bump PyTorch nightly pin past May 21, 2025May 21, 2025
@swolchok
swolchok
commented
May 21, 2025

Copy link


Copy Markdown

ContributorAuthor
looks like we have more breakage if I try to roll forward to the present day. let's go back to at least catching up to April 22
@swolchok
Update
a212e81
[ghstack-poisoned]
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past May 21, 2025Bump PyTorch nightly pin past April 22, 2025May 21, 2025
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
08dca3d
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: ca64709
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past April 22, 2025Bump PyTorch nightly pin past May 21, 2025May 21, 2025
@cccclai
cccclai
commented
May 21, 2025

Copy link


Copy Markdown

Contributor
#10769 is merged as fyi
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past May 21, 2025Bump PyTorch nightly pin past April 22, 2025May 21, 2025
@swolchok
sync c10
089917c
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
660b1bc
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: f50a6df
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
added
the 
release notes: none
Do not include this in the release notes
label
May 21, 2025
@swolchok
swolchok
commented
May 22, 2025

Copy link


Copy Markdown

ContributorAuthor
test-llava-runner-linux seems to be consistently failing with SIGILL :(
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
test-llava-runner-linux seems to be consistently failing with SIGILL :(
Hm, I successfully ran python -m unittest examples.models.llava.test.test_llava with this PR on a Linux box, and Google says that the runners support AVX512, so it's not something simple like that. Trying one more time.
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
if I'm reading correctly, the illegal instruction is a vgf2p8affineqb somewhere in portable_lib. this is an AVX512GFNI instruction which requires at least Ice Lake and the processor in question appears to be Cascade Lake.
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
This instruction does seem to be generated in xnn_f32_vrcopysignc_ukernel__sse2_u16 in my local build as well, I just have a machine handy that can execute it. I guess something in the pin bump must've caused us to start delegating this op to XNNPACK? CC @digantdesai
@swolchok
swolchok
commented
May 23, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
in xnn_f32_vrcopysignc_ukernel__sse2_u16
the symbol name here may or may not be accurate; I just found that XNNPACK has an avx512vnnigfni configuration
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
we are in fact building at least some avx512vnnigfni files per https://github.com/pytorch/executorch/actions/runs/15171458271/job/42808833402?pr=10362
however, this is not new; https://github.com/pytorch/executorch/actions/runs/15218616519/job/42809734929 (a recent trunk run from HUD) also builds them.
either 1) we are delegating some op that we weren't delegating before AND XNNPACK's gating for these GFNI kernels was always broken, 2) bumping the pytorch pin somehow magically caused ExecuTorch's copy of XNNPACK to change, or 3) something else I haven't thought of yet.
@digantdesai
digantdesai
commented
May 24, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
This instruction is likely coming from the QB4W gemm kernel. This kernel is only reachable through ET and not PT. Not sure why it is getting activated on a Cascade Lake CPU on the CI C5, esp on a pin bump. At compile time this kernel compilation is gated by XNNPACK_ENABLE_AVX512VNNIGFNI and at runtime it is selected if cpuinfo_has_x86_avx512vnni() && cpuinfo_has_x86_gfni().
For now you may try disabling these kernels at compile time, to unblock yourself from updating the pin. I can try to take a closer look next week. Like you, the x86 box I have at hand is also Icelake which supports these.
@swolchok
rebase (undo apparently-accidental pin bump only in install_requireme…
b8f6b50
…nts), disable XNNPACK_ENABLE_AVX512VNNIGFNI
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 27, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
6331dee
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: 6c5d261
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchok
swolchok
commented
May 27, 2025

Copy link


Copy Markdown

ContributorAuthor
failures seem to pattern-match to known issues on main. merging.
@swolchok
swolchok
merged commit 4cf5c06
into
mainMay 27, 2025
@swolchok
swolchok
deleted the

gh/swolchok/422/head

branch
May 27, 2025 17:59
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mergennachinmergennachinmergennachin approved these changes
@jackzhxngjackzhxngAwaiting requested review from jackzhxng
@GregoryComerGregoryComerAwaiting requested review from GregoryComer
@JacobSzwejbkaJacobSzwejbkaAwaiting requested review from JacobSzwejbkaJacobSzwejbka is a code owner
@lucylqlucylqAwaiting requested review from lucylq
@digantdesaidigantdesaiAwaiting requested review from digantdesaidigantdesai is a code owner
@mcr229mcr229Awaiting requested review from mcr229

Assignees


No one assigned


Labels

ciflow/trunkCLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.release notes: noneDo not include this in the release notestopic: not user facing

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@swolchok@cccclai@digantdesai@mergennachin@facebook-github-bot

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Bump PyTorch nightly pin past April 22, 2025 - #10362
Merged
swolchok merged 5 commits into
mainfrom 
gh/swolchok/422/head
May 27, 2025
Merged
Bump PyTorch nightly pin past April 22, 2025#10362
swolchok merged 5 commits into
mainfrom 
gh/swolchok/422/head
Conversation
@swolchok
@swolchokswolchok
commented
Apr 22, 2025

Copy link


Copy Markdown

Contributor
@swolchok
Update
298df23
[ghstack-poisoned]
@swolchok
swolchok
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Stack from ghstack (oldest at bottom):
@pytorch-bot
pytorch-botBot
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

🔗 Helpful Links
🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10362
Note: Links to docs will display an error until the docs builds have been completed.
❌ 2 New Failures, 3 Unrelated Failures
As of commit b8f6b50 with merge base 3066463 (image):
NEW FAILURES - The following jobs have failed:
BROKEN TRUNK - The following jobs failed but were present on the merge base:
👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes.
swolchok added a commit
that referenced
this pull request
Apr 22, 2025
@swolchok
Bump PyTorch nightly pin past April 22 2025
eef7ac1
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: 7ef4d5c
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@facebook-github-botfacebook-github-bot
added
the 
CLA Signed
This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.
label
Apr 22, 2025
@swolchok
swolchok
commented
Apr 22, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
"Build documentation / check-urls" failure looks spurious; will rerun when the option becomes available.

ditto "trunk / test-models-linux-aarch64 (qwen2_5, portable, linux.arm64.2xlarge) / linux-job (push)"
@cccclai
cccclai
commented
Apr 22, 2025

Copy link


Copy Markdown

Contributor
Can we hold on bumping? The PR pytorch/pytorch#151436 merged last week causing 50+ tests failing...
@swolchok
swolchok
commented
Apr 23, 2025

Copy link


Copy Markdown

ContributorAuthor
hold on bumping?
Can we hold on bumping? The PR pytorch/pytorch#151436 merged last week causing 50+ tests failing...
@cccclai OK please let me know when we can do this
@cccclai
cccclai
commented
Apr 23, 2025

Copy link


Copy Markdown

Contributor
@tugsbayasgalan is working on resolving the failure.
@cccclaicccclai
mentioned this pull request
Apr 23, 2025
 Merged

@swolchok
swolchok
commented
Apr 25, 2025

Copy link


Copy Markdown

ContributorAuthor
ping @cccclai@tugsbayasgalan
@cccclai
cccclai
commented
May 13, 2025

Copy link


Copy Markdown

Contributor
ping @cccclai@tugsbayasgalan
See discussion in #10319 (comment), hopefully it's resolved soon
@cccclai
cccclai
commented
May 15, 2025

Copy link


Copy Markdown

Contributor
#10769 for the fix.
@swolchok
Update
d23fef6
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
322c879
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: fbc5600
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past April 22 2025Bump PyTorch nightly pin past May 21, 2025May 21, 2025
@swolchok
swolchok
commented
May 21, 2025

Copy link


Copy Markdown

ContributorAuthor
looks like we have more breakage if I try to roll forward to the present day. let's go back to at least catching up to April 22
@swolchok
Update
a212e81
[ghstack-poisoned]
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past May 21, 2025Bump PyTorch nightly pin past April 22, 2025May 21, 2025
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
08dca3d
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: ca64709
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past April 22, 2025Bump PyTorch nightly pin past May 21, 2025May 21, 2025
@cccclai
cccclai
commented
May 21, 2025

Copy link


Copy Markdown

Contributor
#10769 is merged as fyi
@swolchokswolchok
changed the title
Bump PyTorch nightly pin past May 21, 2025Bump PyTorch nightly pin past April 22, 2025May 21, 2025
@swolchok
sync c10
089917c
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 21, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
660b1bc
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: f50a6df
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchokswolchok
added
the 
release notes: none
Do not include this in the release notes
label
May 21, 2025
@swolchok
swolchok
commented
May 22, 2025

Copy link


Copy Markdown

ContributorAuthor
test-llava-runner-linux seems to be consistently failing with SIGILL :(
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
test-llava-runner-linux seems to be consistently failing with SIGILL :(
Hm, I successfully ran python -m unittest examples.models.llava.test.test_llava with this PR on a Linux box, and Google says that the runners support AVX512, so it's not something simple like that. Trying one more time.
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
if I'm reading correctly, the illegal instruction is a vgf2p8affineqb somewhere in portable_lib. this is an AVX512GFNI instruction which requires at least Ice Lake and the processor in question appears to be Cascade Lake.
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
This instruction does seem to be generated in xnn_f32_vrcopysignc_ukernel__sse2_u16 in my local build as well, I just have a machine handy that can execute it. I guess something in the pin bump must've caused us to start delegating this op to XNNPACK? CC @digantdesai
@swolchok
swolchok
commented
May 23, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
in xnn_f32_vrcopysignc_ukernel__sse2_u16
the symbol name here may or may not be accurate; I just found that XNNPACK has an avx512vnnigfni configuration
@swolchok
swolchok
commented
May 23, 2025

Copy link


Copy Markdown

ContributorAuthor
we are in fact building at least some avx512vnnigfni files per https://github.com/pytorch/executorch/actions/runs/15171458271/job/42808833402?pr=10362
however, this is not new; https://github.com/pytorch/executorch/actions/runs/15218616519/job/42809734929 (a recent trunk run from HUD) also builds them.
either 1) we are delegating some op that we weren't delegating before AND XNNPACK's gating for these GFNI kernels was always broken, 2) bumping the pytorch pin somehow magically caused ExecuTorch's copy of XNNPACK to change, or 3) something else I haven't thought of yet.
@digantdesai
digantdesai
commented
May 24, 2025•

edited

Loading

Copy link


Copy Markdown

Contributor
This instruction is likely coming from the QB4W gemm kernel. This kernel is only reachable through ET and not PT. Not sure why it is getting activated on a Cascade Lake CPU on the CI C5, esp on a pin bump. At compile time this kernel compilation is gated by XNNPACK_ENABLE_AVX512VNNIGFNI and at runtime it is selected if cpuinfo_has_x86_avx512vnni() && cpuinfo_has_x86_gfni().
For now you may try disabling these kernels at compile time, to unblock yourself from updating the pin. I can try to take a closer look next week. Like you, the x86 box I have at hand is also Icelake which supports these.
@swolchok
rebase (undo apparently-accidental pin bump only in install_requireme…
b8f6b50
…nts), disable XNNPACK_ENABLE_AVX512VNNIGFNI
[ghstack-poisoned]
swolchok added a commit
that referenced
this pull request
May 27, 2025
@swolchok
Bump PyTorch nightly pin past May 21 2025
6331dee
Found the commit hash from https://hud2.pytorch.org/hud/pytorch/pytorch/nightly/0 .
ghstack-source-id: 6c5d261
ghstack-comment-id: 2822111987
Pull-Request-resolved: #10362
@swolchok
swolchok
commented
May 27, 2025

Copy link


Copy Markdown

ContributorAuthor
failures seem to pattern-match to known issues on main. merging.
@swolchok
swolchok
merged commit 4cf5c06
into
mainMay 27, 2025
@swolchok
swolchok
deleted the

gh/swolchok/422/head

branch
May 27, 2025 17:59
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@mergennachinmergennachinmergennachin approved these changes
@jackzhxngjackzhxngAwaiting requested review from jackzhxng
@GregoryComerGregoryComerAwaiting requested review from GregoryComer
@JacobSzwejbkaJacobSzwejbkaAwaiting requested review from JacobSzwejbkaJacobSzwejbka is a code owner
@lucylqlucylqAwaiting requested review from lucylq
@digantdesaidigantdesaiAwaiting requested review from digantdesaidigantdesai is a code owner
@mcr229mcr229Awaiting requested review from mcr229

Assignees


No one assigned


Labels

ciflow/trunkCLA SignedThis label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed.release notes: noneDo not include this in the release notestopic: not user facing

Projects


None yet


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

5 participants

@swolchok@cccclai@digantdesai@mergennachin@facebook-github-bot