[Backend Tester] Upload test report artifacts in CI job by GregoryComer · Pull Request #13444 · pytorch/executorch

GregoryComer · 2025-08-15T00:09:26Z

Upload the test reports as part of the nightly job. This report contains the test case summary and details, such as accuracy statistics, PTE size, and lowering time.

[ghstack-poisoned]

GregoryComer · 2025-08-15T00:09:28Z

Stack from ghstack (oldest at bottom):

pytorch-bot · 2025-08-15T00:09:30Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13444

📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 7 New Failures, 45 Unrelated Failures

As of commit 4d5f1d0 with merge base 5b743f5 ():

NEW FAILURES - The following jobs have failed:

Build documentation / build (buck2) / Build doc (gh)
At least one of the pre-conditions you specified did not hold
nightly / backend-test-linux (xnnpack_static_int8_per_channel, operators) / linux-job (gh)
test_sqrt_edge_cases_xnnpack_static_int8_per_channel
nightly / backend-test-linux (xnnpack, operators) / linux-job (gh)
test_sqrt_edge_cases_xnnpack
trunk / test-huggingface-optimum-coreml (distilbert|coreml_fp32_gpu|--quantize) / macos-job (gh)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
trunk / test-models-arm-zephyr (add) / linux-job (gh)
RuntimeError: Command docker exec -t abcca9cf1fb9ae4e4b091a2f5ab1936f2f4ce2d333fd9ec9cb2e626085c7141c /exec failed with exit code 1
trunk / test-models-arm-zephyr (mv2) / linux-job (gh)
RuntimeError: Command docker exec -t 83b0ccdbcad892eaf16c8b6ce739906c71b652acf7d876e71183691cb7a93e0a /exec failed with exit code 1
trunk / test-models-arm-zephyr (softmax) / linux-job (gh)
RuntimeError: Command docker exec -t dfe47eb66b86ac17ad53d08d253f94c725fc9922c7dc9c73769c77b21a996d70 /exec failed with exit code 1

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

nightly / backend-test-linux (xnnpack_static_int8_per_channel, models) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
nightly / backend-test-linux (xnnpack, models) / linux-job (gh) (detected as infra flaky with no log or failing log classifier)
trunk / test-huggingface-optimum-coreml (olmo|coreml_fp32_gpu|--quantize) / macos-job (gh) (similar failure)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / unittest / macos / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
pull / unittest-buck / macos / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
pull / unittest-editable / macos / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-coreml-delegate / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
trunk / test-custom-ops-macos (cmake) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-huggingface-optimum-coreml (bert|coreml_fp32_gpu|--quantize) / macos-job (gh) (trunk failure)
trunk / test-huggingface-optimum-coreml (llama3|coreml_fp32_gpu|--quantize) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
trunk / test-huggingface-optimum-coreml (qwen3|coreml_fp32_gpu|--quantize) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
trunk / test-huggingface-optimum-coreml (smollm|coreml_fp32_gpu|--quantize) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-llama-runner-mac (fp32, mps) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-llama-runner-mac (fp32, xnnpack+custom+quantize_kv) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-llama-torchao-lowbit / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (dl3) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (edsr) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (efficient_sam) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (emformer_join) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (emformer_transcribe) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (ic3) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (ic4) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (mobilebert) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (mv2) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (mv3) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (resnet50) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (vit) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-coreml (w2l) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (efficient_sam, portable) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (emformer_join, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (ic4, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (llama, portable) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (llama2, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (llama3_2_vision_encoder, portable) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (mobilebert, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (mv3, portable) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (mv3, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (resnet50, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (vit, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-cpu (w2l, xnnpack-quantization-delegation) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-models-macos-mps / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-selective-build-macos (cmake) / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / test-static-llama-ane / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
trunk / unittest-release / macos / macos-job (gh) (trunk failure)
RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: d911d53 ghstack-comment-id: 3190250291 Pull-Request: #13444

[ghstack-poisoned]

GregoryComer added 30 commits July 17, 2025 17:26

Update
f120e70
[ghstack-poisoned]

Update
0fb85e6
[ghstack-poisoned]

Update
4d8d844
[ghstack-poisoned]

Update
dc12b40
[ghstack-poisoned]

Update
ead0616
[ghstack-poisoned]

Update
0f13676
[ghstack-poisoned]

Update
b0b01f2
[ghstack-poisoned]

Update
8b9c9ef
[ghstack-poisoned]

Update
06bf03a
[ghstack-poisoned]

Update
2f8f49b
[ghstack-poisoned]

Update
8ca7766
[ghstack-poisoned]

Update
bffb95f
[ghstack-poisoned]

Update
d21492b
[ghstack-poisoned]

Update
e2c4ea5
[ghstack-poisoned]

Update
8230848
[ghstack-poisoned]

Update
2a1f564
[ghstack-poisoned]

Update
b35e7b1
[ghstack-poisoned]

Update
5c4c6ce
[ghstack-poisoned]

Update
9397803
[ghstack-poisoned]

Update
9dfeb5a
[ghstack-poisoned]

Update
ff5c4a5
[ghstack-poisoned]

Update
42a5de5
[ghstack-poisoned]

Update
402d8f5
[ghstack-poisoned]

Update
34d3ab3
[ghstack-poisoned]

Update
1105e04
[ghstack-poisoned]

Update
482bd21
[ghstack-poisoned]

Update
ea548b7
[ghstack-poisoned]

Update
4108f54
[ghstack-poisoned]

Update
7ef236b
[ghstack-poisoned]

Update
4a58c9d
[ghstack-poisoned]

GregoryComer added 10 commits August 14, 2025 15:57

Update
871ea46
[ghstack-poisoned]

Update
f7cfc18
[ghstack-poisoned]

Update
1a3844c
[ghstack-poisoned]

Update
d951c8a
[ghstack-poisoned]

Update
fe7e387
[ghstack-poisoned]

Update
0dfca3b
[ghstack-poisoned]

Update
de1ef9a
[ghstack-poisoned]

Update
70fd984
[ghstack-poisoned]

Update
482c469
[ghstack-poisoned]

Update
4d42d47
[ghstack-poisoned]

meta-claBot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 15, 2025

GregoryComer added a commit that referenced this pull request Aug 15, 2025
[Backend Tester] Upload test report artifacts in CI job
977cbfe
ghstack-source-id: d911d53 ghstack-comment-id: 3190250291 Pull-Request: #13444

This was referenced Aug 15, 2025
[Backend Tester] Add nightly CI job for XNNPACK #13390
Merged
[Backend Tester] Mark adaptive avgpool2d as an unsupported portable op #13398
Merged
[Backend Tester] Run Vulkan tests in nightly CI #13445
Merged

GregoryComer added 3 commits August 14, 2025 17:26

Update
ee716bc
[ghstack-poisoned]

Update
ac56137
[ghstack-poisoned]

Update
ad1ddaa
[ghstack-poisoned]

GregoryComer mentioned this pull request Aug 15, 2025
[Backend Tester] Run Core ML tests in nightly CI #13446
Merged

GregoryComerand others added 6 commits August 14, 2025 18:15

Update
02c2e14
[ghstack-poisoned]

Update
5038454
[ghstack-poisoned]

Update
f070aa4
[ghstack-poisoned]

Update
f944c0a
[ghstack-poisoned]

Update
c986013
[ghstack-poisoned]

Update
4d5f1d0
[ghstack-poisoned]

Base automatically changed from gh/GregoryComer/127/head to mainAugust 19, 2025 21:57

GregoryComer requested a review from cccclai as a code owner August 19, 2025 21:57

GregoryComer closed this Aug 27, 2025

Navigation Menu

Uh oh!

[Backend Tester] Upload test report artifacts in CI job - #13444

[Backend Tester] Upload test report artifacts in CI job#13444
GregoryComer wants to merge 237 commits into
mainfrom
gh/GregoryComer/128/head

GregoryComer commented Aug 15, 2025•
edited
Loading

Uh oh!

GregoryComer commented Aug 15, 2025•
edited
Loading

Uh oh!

pytorch-botBot commented Aug 15, 2025•
edited
Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

GregoryComer commented Aug 15, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

GregoryComer commented Aug 15, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-botBot commented Aug 15, 2025• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/13444

❌ 7 New Failures, 45 Unrelated Failures

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

GregoryComer commented Aug 15, 2025•
edited
Loading

GregoryComer commented Aug 15, 2025•
edited
Loading

pytorch-botBot commented Aug 15, 2025•
edited
Loading