Arm backend: Fix sigmoid int16 and int32 flakyness

[oscarandersson8218]

Step vela pin and remove all related pytest.mark_flaky in sigmoid int16 and int32 tests.

cc @digantdesai@freddan80@per@zingo

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10548

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 20 New Failures, 8 Unrelated Failures

As of commit 43cec8e with merge base e1738cc ():

NEW FAILURES - The following jobs have failed:

• pull / android / build-llm-demo / linux-job (gh)
  RuntimeError: Command docker exec -t 5d4e17cdedfcb29e2f1d3f18dceafb38b98b16de079ad299b7cc7c7c91024cde /exec failed with exit code 1
• pull / test-eval_llama-mmlu-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 7d719a78715b3786cc2db3d9005e8324c92beb99613a842190c20a9b016a01a3 /exec failed with exit code 1
• pull / test-eval_llama-wikitext-linux / linux-job (gh)
  RuntimeError: Command docker exec -t d11efbd3ee29e8f6e9a8f612b25388cc38417e0d716ea5dd808c876e7375cca7 /exec failed with exit code 1
• pull / test-llama_runner_eager-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 2ab4123860e8b30f2a15db9e0274b2e6a8a39f40becf214d4b7452a2687bb6af /exec failed with exit code 1
• pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t ab4fc77a9d9f3f9fc75bd41b90e2be3f1ea65022a42a0a55d68df62508e57a55 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 12f54015695b0f24934dcfd6db019d6fdc701e7e73743c8535d0dd721971c3d9 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t f7bf1c4819605d6dffa70d4392ab81c05abd5ae1962ce419d4adffe3439e54d8 /exec failed with exit code 1
• pull / test-llava-runner-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 83d2d00cf49f2c691058d2c8e2673bfee56ed3af3a86041dd841382a5703efb2 /exec failed with exit code 139
• pull / test-models-linux (phi_4_mini, portable, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t 831a5a0f210fe7ceb5d20c244f95b8da0e6d8c76aa9ab7c6ca9f4a62f9a47816 /exec failed with exit code 1
• trunk / test-arm-backend (test_pytest_ops_ethosu_fvp) / linux-job (gh)
  RuntimeError: Command docker exec -t 5e351ca7e133130f77e18482fc7579cb9259371736d26a1e20c49fd8b1773a7e /exec failed with exit code 1
• trunk / test-coreml-delegate / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-linux (bf16, custom, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t 3d9f492305e7708fb4c358bb3b7cd97cd62e116e8b3c0e589e9d03e1bbc6dc29 /exec failed with exit code 1
• trunk / test-llama-runner-linux (bf16, portable, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t ddeecbb162d647acd72359cca99924465ce26d64f1c625effd505ce2ff9d8909 /exec failed with exit code 1
• trunk / test-llama-runner-linux (bf16, portable, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t e7d88ca5528e2ece791484996684db79b1d16a96b4102a0eb93a18f25d0aa2f1 /exec failed with exit code 1
• trunk / test-llama-runner-linux (fp32, portable, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t c310ca364c6ba33ae73a74775b309a02cc63977c9805914384303da174c5c7c1 /exec failed with exit code 1
• trunk / test-llama-runner-linux (fp32, portable, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t 801bc6e77dd82acb5a3998dd4e88648279218fe086221725c5efc77d4d776373 /exec failed with exit code 1
• trunk / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 4d7c8105f0c566b7811f16d9a1935e727046cd3f6d376142d766200774f79253 /exec failed with exit code 1
• trunk / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 689eb0a3e98b3df643845ce5ff90952461a3b29336e0611d897136add082d232 /exec failed with exit code 1
• trunk / test-models-linux-aarch64 (phi_4_mini, portable, linux.arm64.m7g.4xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d28a62903a9f27907e0f12ccc78e054fc9d273f7ce2b7bea2718981475c8a3c6 /exec failed with exit code 1
• trunk / test-models-linux-aarch64 (qwen2_5, portable, linux.arm64.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d433f77868ddfd1cc6579ca44be1cc07aa6a52f30f2c36169443b3c11472f970 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest-editable / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-mac (fp32, mps) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / unittest-release / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• trunk / unittest-release / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[zingo]

@digantdesai this is also steping Vela (e.g. it's newer then #9893 ) Maybe you want to merge both PRs at the same time using the version of Vela from this PR.

EDIT we need to update this PR a bit more @digantdesai do not merge it yet

[oscarandersson8218]

Need to investigate some of the failures.

[oscarandersson8218]

@digantdesai@zingo It's now been resolved and is ready for review again. A few depthwise_conv2d tests started failing in different way than before so had to update those.

[zingo]

@digantdesai we think this is ready and should work now. This will improve INT-16 tables so it will be interesting to get this in.

[facebook-github-bot]

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[zingo]

Hi unfortunately it got a merge conflict and probably needs to be rebased after #10776 was merged.
Sorry about that.

[zingo]

@digantdesai does this work now and can be merged?

[facebook-github-bot]

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[digantdesai]

Yeah looks good internally, feel free to merge. CI failures looks unrelated?

[zingo]

Thanks for the help!

[zingo]

Yes those fails was a geneal llama errors a few days ago that has been fixed.

[oscarandersson8218]

Step vela pin and remove all related pytest.mark_flaky in sigmoid int16 and int32 tests.

cc @digantdesai@freddan80@per@zingo

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10548

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 20 New Failures, 8 Unrelated Failures

As of commit 43cec8e with merge base e1738cc ():

NEW FAILURES - The following jobs have failed:

• pull / android / build-llm-demo / linux-job (gh)
  RuntimeError: Command docker exec -t 5d4e17cdedfcb29e2f1d3f18dceafb38b98b16de079ad299b7cc7c7c91024cde /exec failed with exit code 1
• pull / test-eval_llama-mmlu-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 7d719a78715b3786cc2db3d9005e8324c92beb99613a842190c20a9b016a01a3 /exec failed with exit code 1
• pull / test-eval_llama-wikitext-linux / linux-job (gh)
  RuntimeError: Command docker exec -t d11efbd3ee29e8f6e9a8f612b25388cc38417e0d716ea5dd808c876e7375cca7 /exec failed with exit code 1
• pull / test-llama_runner_eager-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 2ab4123860e8b30f2a15db9e0274b2e6a8a39f40becf214d4b7452a2687bb6af /exec failed with exit code 1
• pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t ab4fc77a9d9f3f9fc75bd41b90e2be3f1ea65022a42a0a55d68df62508e57a55 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 12f54015695b0f24934dcfd6db019d6fdc701e7e73743c8535d0dd721971c3d9 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t f7bf1c4819605d6dffa70d4392ab81c05abd5ae1962ce419d4adffe3439e54d8 /exec failed with exit code 1
• pull / test-llava-runner-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 83d2d00cf49f2c691058d2c8e2673bfee56ed3af3a86041dd841382a5703efb2 /exec failed with exit code 139
• pull / test-models-linux (phi_4_mini, portable, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t 831a5a0f210fe7ceb5d20c244f95b8da0e6d8c76aa9ab7c6ca9f4a62f9a47816 /exec failed with exit code 1
• trunk / test-arm-backend (test_pytest_ops_ethosu_fvp) / linux-job (gh)
  RuntimeError: Command docker exec -t 5e351ca7e133130f77e18482fc7579cb9259371736d26a1e20c49fd8b1773a7e /exec failed with exit code 1
• trunk / test-coreml-delegate / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-linux (bf16, custom, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t 3d9f492305e7708fb4c358bb3b7cd97cd62e116e8b3c0e589e9d03e1bbc6dc29 /exec failed with exit code 1
• trunk / test-llama-runner-linux (bf16, portable, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t ddeecbb162d647acd72359cca99924465ce26d64f1c625effd505ce2ff9d8909 /exec failed with exit code 1
• trunk / test-llama-runner-linux (bf16, portable, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t e7d88ca5528e2ece791484996684db79b1d16a96b4102a0eb93a18f25d0aa2f1 /exec failed with exit code 1
• trunk / test-llama-runner-linux (fp32, portable, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t c310ca364c6ba33ae73a74775b309a02cc63977c9805914384303da174c5c7c1 /exec failed with exit code 1
• trunk / test-llama-runner-linux (fp32, portable, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t 801bc6e77dd82acb5a3998dd4e88648279218fe086221725c5efc77d4d776373 /exec failed with exit code 1
• trunk / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 4d7c8105f0c566b7811f16d9a1935e727046cd3f6d376142d766200774f79253 /exec failed with exit code 1
• trunk / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 689eb0a3e98b3df643845ce5ff90952461a3b29336e0611d897136add082d232 /exec failed with exit code 1
• trunk / test-models-linux-aarch64 (phi_4_mini, portable, linux.arm64.m7g.4xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d28a62903a9f27907e0f12ccc78e054fc9d273f7ce2b7bea2718981475c8a3c6 /exec failed with exit code 1
• trunk / test-models-linux-aarch64 (qwen2_5, portable, linux.arm64.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d433f77868ddfd1cc6579ca44be1cc07aa6a52f30f2c36169443b3c11472f970 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest-editable / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-mac (fp32, mps) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / unittest-release / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• trunk / unittest-release / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[zingo]

@digantdesai this is also steping Vela (e.g. it's newer then #9893 ) Maybe you want to merge both PRs at the same time using the version of Vela from this PR.

EDIT we need to update this PR a bit more @digantdesai do not merge it yet

[oscarandersson8218]

Need to investigate some of the failures.

[oscarandersson8218]

@digantdesai@zingo It's now been resolved and is ready for review again. A few depthwise_conv2d tests started failing in different way than before so had to update those.

[zingo]

@digantdesai we think this is ready and should work now. This will improve INT-16 tables so it will be interesting to get this in.

[facebook-github-bot]

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[zingo]

Hi unfortunately it got a merge conflict and probably needs to be rebased after #10776 was merged.
Sorry about that.

[zingo]

@digantdesai does this work now and can be merged?

[facebook-github-bot]

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[digantdesai]

Yeah looks good internally, feel free to merge. CI failures looks unrelated?

[zingo]

Thanks for the help!

[zingo]

Yes those fails was a geneal llama errors a few days ago that has been fixed.

[oscarandersson8218]

Step vela pin and remove all related pytest.mark_flaky in sigmoid int16 and int32 tests.

cc @digantdesai@freddan80@per@zingo

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10548

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 20 New Failures, 8 Unrelated Failures

As of commit 43cec8e with merge base e1738cc ():

NEW FAILURES - The following jobs have failed:

• pull / android / build-llm-demo / linux-job (gh)
  RuntimeError: Command docker exec -t 5d4e17cdedfcb29e2f1d3f18dceafb38b98b16de079ad299b7cc7c7c91024cde /exec failed with exit code 1
• pull / test-eval_llama-mmlu-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 7d719a78715b3786cc2db3d9005e8324c92beb99613a842190c20a9b016a01a3 /exec failed with exit code 1
• pull / test-eval_llama-wikitext-linux / linux-job (gh)
  RuntimeError: Command docker exec -t d11efbd3ee29e8f6e9a8f612b25388cc38417e0d716ea5dd808c876e7375cca7 /exec failed with exit code 1
• pull / test-llama_runner_eager-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 2ab4123860e8b30f2a15db9e0274b2e6a8a39f40becf214d4b7452a2687bb6af /exec failed with exit code 1
• pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t ab4fc77a9d9f3f9fc75bd41b90e2be3f1ea65022a42a0a55d68df62508e57a55 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 12f54015695b0f24934dcfd6db019d6fdc701e7e73743c8535d0dd721971c3d9 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t f7bf1c4819605d6dffa70d4392ab81c05abd5ae1962ce419d4adffe3439e54d8 /exec failed with exit code 1
• pull / test-llava-runner-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 83d2d00cf49f2c691058d2c8e2673bfee56ed3af3a86041dd841382a5703efb2 /exec failed with exit code 139
• pull / test-models-linux (phi_4_mini, portable, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t 831a5a0f210fe7ceb5d20c244f95b8da0e6d8c76aa9ab7c6ca9f4a62f9a47816 /exec failed with exit code 1
• trunk / test-arm-backend (test_pytest_ops_ethosu_fvp) / linux-job (gh)
  RuntimeError: Command docker exec -t 5e351ca7e133130f77e18482fc7579cb9259371736d26a1e20c49fd8b1773a7e /exec failed with exit code 1
• trunk / test-coreml-delegate / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-linux (bf16, custom, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t 3d9f492305e7708fb4c358bb3b7cd97cd62e116e8b3c0e589e9d03e1bbc6dc29 /exec failed with exit code 1
• trunk / test-llama-runner-linux (bf16, portable, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t ddeecbb162d647acd72359cca99924465ce26d64f1c625effd505ce2ff9d8909 /exec failed with exit code 1
• trunk / test-llama-runner-linux (bf16, portable, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t e7d88ca5528e2ece791484996684db79b1d16a96b4102a0eb93a18f25d0aa2f1 /exec failed with exit code 1
• trunk / test-llama-runner-linux (fp32, portable, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t c310ca364c6ba33ae73a74775b309a02cc63977c9805914384303da174c5c7c1 /exec failed with exit code 1
• trunk / test-llama-runner-linux (fp32, portable, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t 801bc6e77dd82acb5a3998dd4e88648279218fe086221725c5efc77d4d776373 /exec failed with exit code 1
• trunk / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 4d7c8105f0c566b7811f16d9a1935e727046cd3f6d376142d766200774f79253 /exec failed with exit code 1
• trunk / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 689eb0a3e98b3df643845ce5ff90952461a3b29336e0611d897136add082d232 /exec failed with exit code 1
• trunk / test-models-linux-aarch64 (phi_4_mini, portable, linux.arm64.m7g.4xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d28a62903a9f27907e0f12ccc78e054fc9d273f7ce2b7bea2718981475c8a3c6 /exec failed with exit code 1
• trunk / test-models-linux-aarch64 (qwen2_5, portable, linux.arm64.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d433f77868ddfd1cc6579ca44be1cc07aa6a52f30f2c36169443b3c11472f970 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest-editable / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-mac (fp32, mps) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / unittest-release / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• trunk / unittest-release / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[zingo]

@digantdesai this is also steping Vela (e.g. it's newer then #9893 ) Maybe you want to merge both PRs at the same time using the version of Vela from this PR.

EDIT we need to update this PR a bit more @digantdesai do not merge it yet

[oscarandersson8218]

Need to investigate some of the failures.

[oscarandersson8218]

@digantdesai@zingo It's now been resolved and is ready for review again. A few depthwise_conv2d tests started failing in different way than before so had to update those.

[zingo]

@digantdesai we think this is ready and should work now. This will improve INT-16 tables so it will be interesting to get this in.

[facebook-github-bot]

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[zingo]

Hi unfortunately it got a merge conflict and probably needs to be rebased after #10776 was merged.
Sorry about that.

[zingo]

@digantdesai does this work now and can be merged?

[facebook-github-bot]

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[digantdesai]

Yeah looks good internally, feel free to merge. CI failures looks unrelated?

[zingo]

Thanks for the help!

[zingo]

Yes those fails was a geneal llama errors a few days ago that has been fixed.

[oscarandersson8218]

Step vela pin and remove all related pytest.mark_flaky in sigmoid int16 and int32 tests.

cc @digantdesai@freddan80@per@zingo

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10548

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 20 New Failures, 8 Unrelated Failures

As of commit 43cec8e with merge base e1738cc ():

NEW FAILURES - The following jobs have failed:

• pull / android / build-llm-demo / linux-job (gh)
  RuntimeError: Command docker exec -t 5d4e17cdedfcb29e2f1d3f18dceafb38b98b16de079ad299b7cc7c7c91024cde /exec failed with exit code 1
• pull / test-eval_llama-mmlu-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 7d719a78715b3786cc2db3d9005e8324c92beb99613a842190c20a9b016a01a3 /exec failed with exit code 1
• pull / test-eval_llama-wikitext-linux / linux-job (gh)
  RuntimeError: Command docker exec -t d11efbd3ee29e8f6e9a8f612b25388cc38417e0d716ea5dd808c876e7375cca7 /exec failed with exit code 1
• pull / test-llama_runner_eager-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 2ab4123860e8b30f2a15db9e0274b2e6a8a39f40becf214d4b7452a2687bb6af /exec failed with exit code 1
• pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t ab4fc77a9d9f3f9fc75bd41b90e2be3f1ea65022a42a0a55d68df62508e57a55 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 12f54015695b0f24934dcfd6db019d6fdc701e7e73743c8535d0dd721971c3d9 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t f7bf1c4819605d6dffa70d4392ab81c05abd5ae1962ce419d4adffe3439e54d8 /exec failed with exit code 1
• pull / test-llava-runner-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 83d2d00cf49f2c691058d2c8e2673bfee56ed3af3a86041dd841382a5703efb2 /exec failed with exit code 139
• pull / test-models-linux (phi_4_mini, portable, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t 831a5a0f210fe7ceb5d20c244f95b8da0e6d8c76aa9ab7c6ca9f4a62f9a47816 /exec failed with exit code 1
• trunk / test-arm-backend (test_pytest_ops_ethosu_fvp) / linux-job (gh)
  RuntimeError: Command docker exec -t 5e351ca7e133130f77e18482fc7579cb9259371736d26a1e20c49fd8b1773a7e /exec failed with exit code 1
• trunk / test-coreml-delegate / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-linux (bf16, custom, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t 3d9f492305e7708fb4c358bb3b7cd97cd62e116e8b3c0e589e9d03e1bbc6dc29 /exec failed with exit code 1
• trunk / test-llama-runner-linux (bf16, portable, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t ddeecbb162d647acd72359cca99924465ce26d64f1c625effd505ce2ff9d8909 /exec failed with exit code 1
• trunk / test-llama-runner-linux (bf16, portable, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t e7d88ca5528e2ece791484996684db79b1d16a96b4102a0eb93a18f25d0aa2f1 /exec failed with exit code 1
• trunk / test-llama-runner-linux (fp32, portable, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t c310ca364c6ba33ae73a74775b309a02cc63977c9805914384303da174c5c7c1 /exec failed with exit code 1
• trunk / test-llama-runner-linux (fp32, portable, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t 801bc6e77dd82acb5a3998dd4e88648279218fe086221725c5efc77d4d776373 /exec failed with exit code 1
• trunk / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 4d7c8105f0c566b7811f16d9a1935e727046cd3f6d376142d766200774f79253 /exec failed with exit code 1
• trunk / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 689eb0a3e98b3df643845ce5ff90952461a3b29336e0611d897136add082d232 /exec failed with exit code 1
• trunk / test-models-linux-aarch64 (phi_4_mini, portable, linux.arm64.m7g.4xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d28a62903a9f27907e0f12ccc78e054fc9d273f7ce2b7bea2718981475c8a3c6 /exec failed with exit code 1
• trunk / test-models-linux-aarch64 (qwen2_5, portable, linux.arm64.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d433f77868ddfd1cc6579ca44be1cc07aa6a52f30f2c36169443b3c11472f970 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest-editable / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-mac (fp32, mps) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / unittest-release / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• trunk / unittest-release / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[zingo]

@digantdesai this is also steping Vela (e.g. it's newer then #9893 ) Maybe you want to merge both PRs at the same time using the version of Vela from this PR.

EDIT we need to update this PR a bit more @digantdesai do not merge it yet

[oscarandersson8218]

Need to investigate some of the failures.

[oscarandersson8218]

@digantdesai@zingo It's now been resolved and is ready for review again. A few depthwise_conv2d tests started failing in different way than before so had to update those.

[zingo]

@digantdesai we think this is ready and should work now. This will improve INT-16 tables so it will be interesting to get this in.

[facebook-github-bot]

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[zingo]

Hi unfortunately it got a merge conflict and probably needs to be rebased after #10776 was merged.
Sorry about that.

[zingo]

@digantdesai does this work now and can be merged?

[facebook-github-bot]

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[digantdesai]

Yeah looks good internally, feel free to merge. CI failures looks unrelated?

[zingo]

Thanks for the help!

[zingo]

Yes those fails was a geneal llama errors a few days ago that has been fixed.

[oscarandersson8218]

Step vela pin and remove all related pytest.mark_flaky in sigmoid int16 and int32 tests.

cc @digantdesai@freddan80@per@zingo

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10548

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 20 New Failures, 8 Unrelated Failures

As of commit 43cec8e with merge base e1738cc ():

NEW FAILURES - The following jobs have failed:

• pull / android / build-llm-demo / linux-job (gh)
  RuntimeError: Command docker exec -t 5d4e17cdedfcb29e2f1d3f18dceafb38b98b16de079ad299b7cc7c7c91024cde /exec failed with exit code 1
• pull / test-eval_llama-mmlu-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 7d719a78715b3786cc2db3d9005e8324c92beb99613a842190c20a9b016a01a3 /exec failed with exit code 1
• pull / test-eval_llama-wikitext-linux / linux-job (gh)
  RuntimeError: Command docker exec -t d11efbd3ee29e8f6e9a8f612b25388cc38417e0d716ea5dd808c876e7375cca7 /exec failed with exit code 1
• pull / test-llama_runner_eager-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 2ab4123860e8b30f2a15db9e0274b2e6a8a39f40becf214d4b7452a2687bb6af /exec failed with exit code 1
• pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t ab4fc77a9d9f3f9fc75bd41b90e2be3f1ea65022a42a0a55d68df62508e57a55 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 12f54015695b0f24934dcfd6db019d6fdc701e7e73743c8535d0dd721971c3d9 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t f7bf1c4819605d6dffa70d4392ab81c05abd5ae1962ce419d4adffe3439e54d8 /exec failed with exit code 1
• pull / test-llava-runner-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 83d2d00cf49f2c691058d2c8e2673bfee56ed3af3a86041dd841382a5703efb2 /exec failed with exit code 139
• pull / test-models-linux (phi_4_mini, portable, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t 831a5a0f210fe7ceb5d20c244f95b8da0e6d8c76aa9ab7c6ca9f4a62f9a47816 /exec failed with exit code 1
• trunk / test-arm-backend (test_pytest_ops_ethosu_fvp) / linux-job (gh)
  RuntimeError: Command docker exec -t 5e351ca7e133130f77e18482fc7579cb9259371736d26a1e20c49fd8b1773a7e /exec failed with exit code 1
• trunk / test-coreml-delegate / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-linux (bf16, custom, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t 3d9f492305e7708fb4c358bb3b7cd97cd62e116e8b3c0e589e9d03e1bbc6dc29 /exec failed with exit code 1
• trunk / test-llama-runner-linux (bf16, portable, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t ddeecbb162d647acd72359cca99924465ce26d64f1c625effd505ce2ff9d8909 /exec failed with exit code 1
• trunk / test-llama-runner-linux (bf16, portable, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t e7d88ca5528e2ece791484996684db79b1d16a96b4102a0eb93a18f25d0aa2f1 /exec failed with exit code 1
• trunk / test-llama-runner-linux (fp32, portable, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t c310ca364c6ba33ae73a74775b309a02cc63977c9805914384303da174c5c7c1 /exec failed with exit code 1
• trunk / test-llama-runner-linux (fp32, portable, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t 801bc6e77dd82acb5a3998dd4e88648279218fe086221725c5efc77d4d776373 /exec failed with exit code 1
• trunk / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 4d7c8105f0c566b7811f16d9a1935e727046cd3f6d376142d766200774f79253 /exec failed with exit code 1
• trunk / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 689eb0a3e98b3df643845ce5ff90952461a3b29336e0611d897136add082d232 /exec failed with exit code 1
• trunk / test-models-linux-aarch64 (phi_4_mini, portable, linux.arm64.m7g.4xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d28a62903a9f27907e0f12ccc78e054fc9d273f7ce2b7bea2718981475c8a3c6 /exec failed with exit code 1
• trunk / test-models-linux-aarch64 (qwen2_5, portable, linux.arm64.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d433f77868ddfd1cc6579ca44be1cc07aa6a52f30f2c36169443b3c11472f970 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest-editable / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-mac (fp32, mps) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / unittest-release / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• trunk / unittest-release / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[zingo]

@digantdesai this is also steping Vela (e.g. it's newer then #9893 ) Maybe you want to merge both PRs at the same time using the version of Vela from this PR.

EDIT we need to update this PR a bit more @digantdesai do not merge it yet

[oscarandersson8218]

Need to investigate some of the failures.

[oscarandersson8218]

@digantdesai@zingo It's now been resolved and is ready for review again. A few depthwise_conv2d tests started failing in different way than before so had to update those.

[zingo]

@digantdesai we think this is ready and should work now. This will improve INT-16 tables so it will be interesting to get this in.

[facebook-github-bot]

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[zingo]

Hi unfortunately it got a merge conflict and probably needs to be rebased after #10776 was merged.
Sorry about that.

[zingo]

@digantdesai does this work now and can be merged?

[facebook-github-bot]

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[digantdesai]

Yeah looks good internally, feel free to merge. CI failures looks unrelated?

[zingo]

Thanks for the help!

[zingo]

Yes those fails was a geneal llama errors a few days ago that has been fixed.

[oscarandersson8218]

Step vela pin and remove all related pytest.mark_flaky in sigmoid int16 and int32 tests.

cc @digantdesai@freddan80@per@zingo

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10548

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 20 New Failures, 8 Unrelated Failures

As of commit 43cec8e with merge base e1738cc ():

NEW FAILURES - The following jobs have failed:

• pull / android / build-llm-demo / linux-job (gh)
  RuntimeError: Command docker exec -t 5d4e17cdedfcb29e2f1d3f18dceafb38b98b16de079ad299b7cc7c7c91024cde /exec failed with exit code 1
• pull / test-eval_llama-mmlu-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 7d719a78715b3786cc2db3d9005e8324c92beb99613a842190c20a9b016a01a3 /exec failed with exit code 1
• pull / test-eval_llama-wikitext-linux / linux-job (gh)
  RuntimeError: Command docker exec -t d11efbd3ee29e8f6e9a8f612b25388cc38417e0d716ea5dd808c876e7375cca7 /exec failed with exit code 1
• pull / test-llama_runner_eager-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 2ab4123860e8b30f2a15db9e0274b2e6a8a39f40becf214d4b7452a2687bb6af /exec failed with exit code 1
• pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t ab4fc77a9d9f3f9fc75bd41b90e2be3f1ea65022a42a0a55d68df62508e57a55 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 12f54015695b0f24934dcfd6db019d6fdc701e7e73743c8535d0dd721971c3d9 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t f7bf1c4819605d6dffa70d4392ab81c05abd5ae1962ce419d4adffe3439e54d8 /exec failed with exit code 1
• pull / test-llava-runner-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 83d2d00cf49f2c691058d2c8e2673bfee56ed3af3a86041dd841382a5703efb2 /exec failed with exit code 139
• pull / test-models-linux (phi_4_mini, portable, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t 831a5a0f210fe7ceb5d20c244f95b8da0e6d8c76aa9ab7c6ca9f4a62f9a47816 /exec failed with exit code 1
• trunk / test-arm-backend (test_pytest_ops_ethosu_fvp) / linux-job (gh)
  RuntimeError: Command docker exec -t 5e351ca7e133130f77e18482fc7579cb9259371736d26a1e20c49fd8b1773a7e /exec failed with exit code 1
• trunk / test-coreml-delegate / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-linux (bf16, custom, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t 3d9f492305e7708fb4c358bb3b7cd97cd62e116e8b3c0e589e9d03e1bbc6dc29 /exec failed with exit code 1
• trunk / test-llama-runner-linux (bf16, portable, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t ddeecbb162d647acd72359cca99924465ce26d64f1c625effd505ce2ff9d8909 /exec failed with exit code 1
• trunk / test-llama-runner-linux (bf16, portable, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t e7d88ca5528e2ece791484996684db79b1d16a96b4102a0eb93a18f25d0aa2f1 /exec failed with exit code 1
• trunk / test-llama-runner-linux (fp32, portable, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t c310ca364c6ba33ae73a74775b309a02cc63977c9805914384303da174c5c7c1 /exec failed with exit code 1
• trunk / test-llama-runner-linux (fp32, portable, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t 801bc6e77dd82acb5a3998dd4e88648279218fe086221725c5efc77d4d776373 /exec failed with exit code 1
• trunk / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 4d7c8105f0c566b7811f16d9a1935e727046cd3f6d376142d766200774f79253 /exec failed with exit code 1
• trunk / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 689eb0a3e98b3df643845ce5ff90952461a3b29336e0611d897136add082d232 /exec failed with exit code 1
• trunk / test-models-linux-aarch64 (phi_4_mini, portable, linux.arm64.m7g.4xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d28a62903a9f27907e0f12ccc78e054fc9d273f7ce2b7bea2718981475c8a3c6 /exec failed with exit code 1
• trunk / test-models-linux-aarch64 (qwen2_5, portable, linux.arm64.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d433f77868ddfd1cc6579ca44be1cc07aa6a52f30f2c36169443b3c11472f970 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest-editable / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-mac (fp32, mps) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / unittest-release / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• trunk / unittest-release / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[zingo]

@digantdesai this is also steping Vela (e.g. it's newer then #9893 ) Maybe you want to merge both PRs at the same time using the version of Vela from this PR.

EDIT we need to update this PR a bit more @digantdesai do not merge it yet

[oscarandersson8218]

Need to investigate some of the failures.

[oscarandersson8218]

@digantdesai@zingo It's now been resolved and is ready for review again. A few depthwise_conv2d tests started failing in different way than before so had to update those.

[zingo]

@digantdesai we think this is ready and should work now. This will improve INT-16 tables so it will be interesting to get this in.

[facebook-github-bot]

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[zingo]

Hi unfortunately it got a merge conflict and probably needs to be rebased after #10776 was merged.
Sorry about that.

[zingo]

@digantdesai does this work now and can be merged?

[facebook-github-bot]

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[digantdesai]

Yeah looks good internally, feel free to merge. CI failures looks unrelated?

[zingo]

Thanks for the help!

[zingo]

Yes those fails was a geneal llama errors a few days ago that has been fixed.

[oscarandersson8218]

Step vela pin and remove all related pytest.mark_flaky in sigmoid int16 and int32 tests.

cc @digantdesai@freddan80@per@zingo

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10548

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 20 New Failures, 8 Unrelated Failures

As of commit 43cec8e with merge base e1738cc ():

NEW FAILURES - The following jobs have failed:

• pull / android / build-llm-demo / linux-job (gh)
  RuntimeError: Command docker exec -t 5d4e17cdedfcb29e2f1d3f18dceafb38b98b16de079ad299b7cc7c7c91024cde /exec failed with exit code 1
• pull / test-eval_llama-mmlu-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 7d719a78715b3786cc2db3d9005e8324c92beb99613a842190c20a9b016a01a3 /exec failed with exit code 1
• pull / test-eval_llama-wikitext-linux / linux-job (gh)
  RuntimeError: Command docker exec -t d11efbd3ee29e8f6e9a8f612b25388cc38417e0d716ea5dd808c876e7375cca7 /exec failed with exit code 1
• pull / test-llama_runner_eager-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 2ab4123860e8b30f2a15db9e0274b2e6a8a39f40becf214d4b7452a2687bb6af /exec failed with exit code 1
• pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t ab4fc77a9d9f3f9fc75bd41b90e2be3f1ea65022a42a0a55d68df62508e57a55 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 12f54015695b0f24934dcfd6db019d6fdc701e7e73743c8535d0dd721971c3d9 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t f7bf1c4819605d6dffa70d4392ab81c05abd5ae1962ce419d4adffe3439e54d8 /exec failed with exit code 1
• pull / test-llava-runner-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 83d2d00cf49f2c691058d2c8e2673bfee56ed3af3a86041dd841382a5703efb2 /exec failed with exit code 139
• pull / test-models-linux (phi_4_mini, portable, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t 831a5a0f210fe7ceb5d20c244f95b8da0e6d8c76aa9ab7c6ca9f4a62f9a47816 /exec failed with exit code 1
• trunk / test-arm-backend (test_pytest_ops_ethosu_fvp) / linux-job (gh)
  RuntimeError: Command docker exec -t 5e351ca7e133130f77e18482fc7579cb9259371736d26a1e20c49fd8b1773a7e /exec failed with exit code 1
• trunk / test-coreml-delegate / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-linux (bf16, custom, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t 3d9f492305e7708fb4c358bb3b7cd97cd62e116e8b3c0e589e9d03e1bbc6dc29 /exec failed with exit code 1
• trunk / test-llama-runner-linux (bf16, portable, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t ddeecbb162d647acd72359cca99924465ce26d64f1c625effd505ce2ff9d8909 /exec failed with exit code 1
• trunk / test-llama-runner-linux (bf16, portable, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t e7d88ca5528e2ece791484996684db79b1d16a96b4102a0eb93a18f25d0aa2f1 /exec failed with exit code 1
• trunk / test-llama-runner-linux (fp32, portable, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t c310ca364c6ba33ae73a74775b309a02cc63977c9805914384303da174c5c7c1 /exec failed with exit code 1
• trunk / test-llama-runner-linux (fp32, portable, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t 801bc6e77dd82acb5a3998dd4e88648279218fe086221725c5efc77d4d776373 /exec failed with exit code 1
• trunk / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 4d7c8105f0c566b7811f16d9a1935e727046cd3f6d376142d766200774f79253 /exec failed with exit code 1
• trunk / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 689eb0a3e98b3df643845ce5ff90952461a3b29336e0611d897136add082d232 /exec failed with exit code 1
• trunk / test-models-linux-aarch64 (phi_4_mini, portable, linux.arm64.m7g.4xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d28a62903a9f27907e0f12ccc78e054fc9d273f7ce2b7bea2718981475c8a3c6 /exec failed with exit code 1
• trunk / test-models-linux-aarch64 (qwen2_5, portable, linux.arm64.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d433f77868ddfd1cc6579ca44be1cc07aa6a52f30f2c36169443b3c11472f970 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest-editable / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-mac (fp32, mps) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / unittest-release / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• trunk / unittest-release / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[zingo]

@digantdesai this is also steping Vela (e.g. it's newer then #9893 ) Maybe you want to merge both PRs at the same time using the version of Vela from this PR.

EDIT we need to update this PR a bit more @digantdesai do not merge it yet

[oscarandersson8218]

Need to investigate some of the failures.

[oscarandersson8218]

@digantdesai@zingo It's now been resolved and is ready for review again. A few depthwise_conv2d tests started failing in different way than before so had to update those.

[zingo]

@digantdesai we think this is ready and should work now. This will improve INT-16 tables so it will be interesting to get this in.

[facebook-github-bot]

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[zingo]

Hi unfortunately it got a merge conflict and probably needs to be rebased after #10776 was merged.
Sorry about that.

[zingo]

@digantdesai does this work now and can be merged?

[facebook-github-bot]

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[digantdesai]

Yeah looks good internally, feel free to merge. CI failures looks unrelated?

[zingo]

Thanks for the help!

[zingo]

Yes those fails was a geneal llama errors a few days ago that has been fixed.

[oscarandersson8218]

Step vela pin and remove all related pytest.mark_flaky in sigmoid int16 and int32 tests.

cc @digantdesai@freddan80@per@zingo

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/10548

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 20 New Failures, 8 Unrelated Failures

As of commit 43cec8e with merge base e1738cc ():

NEW FAILURES - The following jobs have failed:

• pull / android / build-llm-demo / linux-job (gh)
  RuntimeError: Command docker exec -t 5d4e17cdedfcb29e2f1d3f18dceafb38b98b16de079ad299b7cc7c7c91024cde /exec failed with exit code 1
• pull / test-eval_llama-mmlu-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 7d719a78715b3786cc2db3d9005e8324c92beb99613a842190c20a9b016a01a3 /exec failed with exit code 1
• pull / test-eval_llama-wikitext-linux / linux-job (gh)
  RuntimeError: Command docker exec -t d11efbd3ee29e8f6e9a8f612b25388cc38417e0d716ea5dd808c876e7375cca7 /exec failed with exit code 1
• pull / test-llama_runner_eager-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 2ab4123860e8b30f2a15db9e0274b2e6a8a39f40becf214d4b7452a2687bb6af /exec failed with exit code 1
• pull / test-llama-runner-linux (bf16, custom, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t ab4fc77a9d9f3f9fc75bd41b90e2be3f1ea65022a42a0a55d68df62508e57a55 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 12f54015695b0f24934dcfd6db019d6fdc701e7e73743c8535d0dd721971c3d9 /exec failed with exit code 1
• pull / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t f7bf1c4819605d6dffa70d4392ab81c05abd5ae1962ce419d4adffe3439e54d8 /exec failed with exit code 1
• pull / test-llava-runner-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 83d2d00cf49f2c691058d2c8e2673bfee56ed3af3a86041dd841382a5703efb2 /exec failed with exit code 139
• pull / test-models-linux (phi_4_mini, portable, linux.4xlarge.memory) / linux-job (gh)
  RuntimeError: Command docker exec -t 831a5a0f210fe7ceb5d20c244f95b8da0e6d8c76aa9ab7c6ca9f4a62f9a47816 /exec failed with exit code 1
• trunk / test-arm-backend (test_pytest_ops_ethosu_fvp) / linux-job (gh)
  RuntimeError: Command docker exec -t 5e351ca7e133130f77e18482fc7579cb9259371736d26a1e20c49fd8b1773a7e /exec failed with exit code 1
• trunk / test-coreml-delegate / macos-job (gh)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-linux (bf16, custom, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t 3d9f492305e7708fb4c358bb3b7cd97cd62e116e8b3c0e589e9d03e1bbc6dc29 /exec failed with exit code 1
• trunk / test-llama-runner-linux (bf16, portable, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t ddeecbb162d647acd72359cca99924465ce26d64f1c625effd505ce2ff9d8909 /exec failed with exit code 1
• trunk / test-llama-runner-linux (bf16, portable, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t e7d88ca5528e2ece791484996684db79b1d16a96b4102a0eb93a18f25d0aa2f1 /exec failed with exit code 1
• trunk / test-llama-runner-linux (fp32, portable, linux.2xlarge, executorch-ubuntu-22.04-clang12) / linux-job (gh)
  RuntimeError: Command docker exec -t c310ca364c6ba33ae73a74775b309a02cc63977c9805914384303da174c5c7c1 /exec failed with exit code 1
• trunk / test-llama-runner-linux (fp32, portable, linux.arm64.2xlarge, executorch-ubuntu-22.04-gcc11-aarch64) / linux-job (gh)
  RuntimeError: Command docker exec -t 801bc6e77dd82acb5a3998dd4e88648279218fe086221725c5efc77d4d776373 /exec failed with exit code 1
• trunk / test-llama-runner-qnn-linux (fp32, qnn_16a16w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 4d7c8105f0c566b7811f16d9a1935e727046cd3f6d376142d766200774f79253 /exec failed with exit code 1
• trunk / test-llama-runner-qnn-linux (fp32, qnn_8a8w, qnn) / linux-job (gh)
  RuntimeError: Command docker exec -t 689eb0a3e98b3df643845ce5ff90952461a3b29336e0611d897136add082d232 /exec failed with exit code 1
• trunk / test-models-linux-aarch64 (phi_4_mini, portable, linux.arm64.m7g.4xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d28a62903a9f27907e0f12ccc78e054fc9d273f7ce2b7bea2718981475c8a3c6 /exec failed with exit code 1
• trunk / test-models-linux-aarch64 (qwen2_5, portable, linux.arm64.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t d433f77868ddfd1cc6579ca44be1cc07aa6a52f30f2c36169443b3c11472f970 /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest-editable / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• pull / unittest-editable / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-llama-runner-mac (fp32, mps) / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / unittest-release / linux / linux-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts
• trunk / unittest-release / macos / macos-job (gh) (trunk failure)
  examples/models/llama/tests/test_export_llama_lib.py::ExportLlamaLibTest::test_has_expected_ops_and_op_counts

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[zingo]

@digantdesai this is also steping Vela (e.g. it's newer then #9893 ) Maybe you want to merge both PRs at the same time using the version of Vela from this PR.

EDIT we need to update this PR a bit more @digantdesai do not merge it yet

[oscarandersson8218]

Need to investigate some of the failures.

[oscarandersson8218]

@digantdesai@zingo It's now been resolved and is ready for review again. A few depthwise_conv2d tests started failing in different way than before so had to update those.

[zingo]

@digantdesai we think this is ready and should work now. This will improve INT-16 tables so it will be interesting to get this in.

[facebook-github-bot]

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[zingo]

Hi unfortunately it got a merge conflict and probably needs to be rebased after #10776 was merged.
Sorry about that.

[zingo]

@digantdesai does this work now and can be merged?

[facebook-github-bot]

@digantdesai has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[digantdesai]

Yeah looks good internally, feel free to merge. CI failures looks unrelated?

[zingo]

Thanks for the help!

[zingo]

Yes those fails was a geneal llama errors a few days ago that has been fixed.