Build flatc for the host instead of the target platform

[jathu]

Summary

• FixesFlatc should be built for the host rather than the target when cross-compiling #7260
  • We use flatc as part of the build step to generate some files from FlatBuffer definitions. This implies the tool runs on the host. However, we currently build flatc as part of the main build — which propagates the target CXX flags to flatc. Thus, flatc gets built targeting the target platform. To ensure flatc is built for the host, we can use include flatc as an ExternalProject. This does not propagate the CXX flags
• Some targets implicitly depended on flatc, we now make that requirement explicit
• We currently spread the FLATC_EXECUTABLE defaulting across the project. Let's just centralize this at the root

Test plan

$ ./install_executorch.sh
# Previously flatc was built against the Android target, now they target the host
$ ./build/build_android_library.sh
$ file /Users/jathu/executorch/cmake-out-android-arm64-v8a/third-party/flatbuffers/flatc
/Users/jathu/executorch/cmake-out-android-arm64-v8a/third-party/flatbuffers/flatc: Mach-O 64-bit executable arm64
# Apple builds work as usual, but they use flatc from pip
$ ./build/build_apple_frameworks.sh

cc @larryliu0820@lucylq@swolchok@dbort

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9077

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Cancelled Job, 2 Unrelated Failures

As of commit 4faa443 with merge base cb3ec19 ():

NEW FAILURE - The following job has failed:

• trunk / test-static-llama-ane / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

CANCELLED JOB - The following job was cancelled. Please retry:

• trunk / test-models-macos (cmake, linear, xnnpack-quantization-delegation, macos-m1-stable, 90) / macos-job (gh)
  ##[error]The operation was canceled.

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• trunk / test-arm-reference-delegation / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Apple / build-benchmark-app / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 65

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[swolchok]

looks great, just the one minor comment

[mergennachin]

@jathu I added 'ciflow/trunk' label, which triggers all trunk jobs

Looks like there are a bunch of CI failures

Please fix them.

[jathu]

These failing tests are unrelated to this diff:

• build-benchmark-app / macos-job — fatal error: 'omp.h' file not found is occurring in main
• test-arm-reference-delegation / linux-job — occurring in main
• test-models-macos (cmake, linear, xnnpack-quantization-delegation, macos-m1-stable, 90) / macos-job — similar omp.h problem as above
• test-static-llama-ane / macos-job — incorrect python version issue also occurring in main

[swolchok]

Now seeing this:

$ cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DEXECUTORCH_BUILD_KERNELS_CUSTOM=ON \
-DEXECUTORCH_BUILD_KERNELS_CUSTOM_AOT=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-DCMAKE_BUILD_TYPE=Release \
-DEXECUTORCH_ENABLE_LOGGING=ON \
-Bcmake-out -GNinja
$ cmake --build cmake-out
ninja: error: 'third-party/flatbuffers/flatc', needed by 'schema/include/executorch/schema/program_generated.h', missing and no known rule to make it

Repros on this commit and not the previous one.

[swolchok]

Ninja is extra-picky about dependencies. #9246 fixes

[jathu]

Summary

• FixesFlatc should be built for the host rather than the target when cross-compiling #7260
  • We use flatc as part of the build step to generate some files from FlatBuffer definitions. This implies the tool runs on the host. However, we currently build flatc as part of the main build — which propagates the target CXX flags to flatc. Thus, flatc gets built targeting the target platform. To ensure flatc is built for the host, we can use include flatc as an ExternalProject. This does not propagate the CXX flags
• Some targets implicitly depended on flatc, we now make that requirement explicit
• We currently spread the FLATC_EXECUTABLE defaulting across the project. Let's just centralize this at the root

Test plan

$ ./install_executorch.sh
# Previously flatc was built against the Android target, now they target the host
$ ./build/build_android_library.sh
$ file /Users/jathu/executorch/cmake-out-android-arm64-v8a/third-party/flatbuffers/flatc
/Users/jathu/executorch/cmake-out-android-arm64-v8a/third-party/flatbuffers/flatc: Mach-O 64-bit executable arm64
# Apple builds work as usual, but they use flatc from pip
$ ./build/build_apple_frameworks.sh

cc @larryliu0820@lucylq@swolchok@dbort

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9077

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Cancelled Job, 2 Unrelated Failures

As of commit 4faa443 with merge base cb3ec19 ():

NEW FAILURE - The following job has failed:

• trunk / test-static-llama-ane / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

CANCELLED JOB - The following job was cancelled. Please retry:

• trunk / test-models-macos (cmake, linear, xnnpack-quantization-delegation, macos-m1-stable, 90) / macos-job (gh)
  ##[error]The operation was canceled.

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• trunk / test-arm-reference-delegation / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Apple / build-benchmark-app / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 65

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[swolchok]

looks great, just the one minor comment

[mergennachin]

@jathu I added 'ciflow/trunk' label, which triggers all trunk jobs

Looks like there are a bunch of CI failures

Please fix them.

[jathu]

These failing tests are unrelated to this diff:

• build-benchmark-app / macos-job — fatal error: 'omp.h' file not found is occurring in main
• test-arm-reference-delegation / linux-job — occurring in main
• test-models-macos (cmake, linear, xnnpack-quantization-delegation, macos-m1-stable, 90) / macos-job — similar omp.h problem as above
• test-static-llama-ane / macos-job — incorrect python version issue also occurring in main

[swolchok]

Now seeing this:

$ cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DEXECUTORCH_BUILD_KERNELS_CUSTOM=ON \
-DEXECUTORCH_BUILD_KERNELS_CUSTOM_AOT=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-DCMAKE_BUILD_TYPE=Release \
-DEXECUTORCH_ENABLE_LOGGING=ON \
-Bcmake-out -GNinja
$ cmake --build cmake-out
ninja: error: 'third-party/flatbuffers/flatc', needed by 'schema/include/executorch/schema/program_generated.h', missing and no known rule to make it

Repros on this commit and not the previous one.

[swolchok]

Ninja is extra-picky about dependencies. #9246 fixes

[jathu]

Summary

• FixesFlatc should be built for the host rather than the target when cross-compiling #7260
  • We use flatc as part of the build step to generate some files from FlatBuffer definitions. This implies the tool runs on the host. However, we currently build flatc as part of the main build — which propagates the target CXX flags to flatc. Thus, flatc gets built targeting the target platform. To ensure flatc is built for the host, we can use include flatc as an ExternalProject. This does not propagate the CXX flags
• Some targets implicitly depended on flatc, we now make that requirement explicit
• We currently spread the FLATC_EXECUTABLE defaulting across the project. Let's just centralize this at the root

Test plan

$ ./install_executorch.sh
# Previously flatc was built against the Android target, now they target the host
$ ./build/build_android_library.sh
$ file /Users/jathu/executorch/cmake-out-android-arm64-v8a/third-party/flatbuffers/flatc
/Users/jathu/executorch/cmake-out-android-arm64-v8a/third-party/flatbuffers/flatc: Mach-O 64-bit executable arm64
# Apple builds work as usual, but they use flatc from pip
$ ./build/build_apple_frameworks.sh

cc @larryliu0820@lucylq@swolchok@dbort

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9077

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Cancelled Job, 2 Unrelated Failures

As of commit 4faa443 with merge base cb3ec19 ():

NEW FAILURE - The following job has failed:

• trunk / test-static-llama-ane / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

CANCELLED JOB - The following job was cancelled. Please retry:

• trunk / test-models-macos (cmake, linear, xnnpack-quantization-delegation, macos-m1-stable, 90) / macos-job (gh)
  ##[error]The operation was canceled.

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• trunk / test-arm-reference-delegation / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Apple / build-benchmark-app / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 65

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[swolchok]

looks great, just the one minor comment

[mergennachin]

@jathu I added 'ciflow/trunk' label, which triggers all trunk jobs

Looks like there are a bunch of CI failures

Please fix them.

[jathu]

These failing tests are unrelated to this diff:

• build-benchmark-app / macos-job — fatal error: 'omp.h' file not found is occurring in main
• test-arm-reference-delegation / linux-job — occurring in main
• test-models-macos (cmake, linear, xnnpack-quantization-delegation, macos-m1-stable, 90) / macos-job — similar omp.h problem as above
• test-static-llama-ane / macos-job — incorrect python version issue also occurring in main

[swolchok]

Now seeing this:

$ cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DEXECUTORCH_BUILD_KERNELS_CUSTOM=ON \
-DEXECUTORCH_BUILD_KERNELS_CUSTOM_AOT=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-DCMAKE_BUILD_TYPE=Release \
-DEXECUTORCH_ENABLE_LOGGING=ON \
-Bcmake-out -GNinja
$ cmake --build cmake-out
ninja: error: 'third-party/flatbuffers/flatc', needed by 'schema/include/executorch/schema/program_generated.h', missing and no known rule to make it

Repros on this commit and not the previous one.

[swolchok]

Ninja is extra-picky about dependencies. #9246 fixes

[jathu]

Summary

• FixesFlatc should be built for the host rather than the target when cross-compiling #7260
  • We use flatc as part of the build step to generate some files from FlatBuffer definitions. This implies the tool runs on the host. However, we currently build flatc as part of the main build — which propagates the target CXX flags to flatc. Thus, flatc gets built targeting the target platform. To ensure flatc is built for the host, we can use include flatc as an ExternalProject. This does not propagate the CXX flags
• Some targets implicitly depended on flatc, we now make that requirement explicit
• We currently spread the FLATC_EXECUTABLE defaulting across the project. Let's just centralize this at the root

Test plan

$ ./install_executorch.sh
# Previously flatc was built against the Android target, now they target the host
$ ./build/build_android_library.sh
$ file /Users/jathu/executorch/cmake-out-android-arm64-v8a/third-party/flatbuffers/flatc
/Users/jathu/executorch/cmake-out-android-arm64-v8a/third-party/flatbuffers/flatc: Mach-O 64-bit executable arm64
# Apple builds work as usual, but they use flatc from pip
$ ./build/build_apple_frameworks.sh

cc @larryliu0820@lucylq@swolchok@dbort

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9077

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Cancelled Job, 2 Unrelated Failures

As of commit 4faa443 with merge base cb3ec19 ():

NEW FAILURE - The following job has failed:

• trunk / test-static-llama-ane / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

CANCELLED JOB - The following job was cancelled. Please retry:

• trunk / test-models-macos (cmake, linear, xnnpack-quantization-delegation, macos-m1-stable, 90) / macos-job (gh)
  ##[error]The operation was canceled.

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• trunk / test-arm-reference-delegation / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Apple / build-benchmark-app / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 65

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[swolchok]

looks great, just the one minor comment

[mergennachin]

@jathu I added 'ciflow/trunk' label, which triggers all trunk jobs

Looks like there are a bunch of CI failures

Please fix them.

[jathu]

These failing tests are unrelated to this diff:

• build-benchmark-app / macos-job — fatal error: 'omp.h' file not found is occurring in main
• test-arm-reference-delegation / linux-job — occurring in main
• test-models-macos (cmake, linear, xnnpack-quantization-delegation, macos-m1-stable, 90) / macos-job — similar omp.h problem as above
• test-static-llama-ane / macos-job — incorrect python version issue also occurring in main

[swolchok]

Now seeing this:

$ cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DEXECUTORCH_BUILD_KERNELS_CUSTOM=ON \
-DEXECUTORCH_BUILD_KERNELS_CUSTOM_AOT=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-DCMAKE_BUILD_TYPE=Release \
-DEXECUTORCH_ENABLE_LOGGING=ON \
-Bcmake-out -GNinja
$ cmake --build cmake-out
ninja: error: 'third-party/flatbuffers/flatc', needed by 'schema/include/executorch/schema/program_generated.h', missing and no known rule to make it

Repros on this commit and not the previous one.

[swolchok]

Ninja is extra-picky about dependencies. #9246 fixes

[jathu]

Summary

• FixesFlatc should be built for the host rather than the target when cross-compiling #7260
  • We use flatc as part of the build step to generate some files from FlatBuffer definitions. This implies the tool runs on the host. However, we currently build flatc as part of the main build — which propagates the target CXX flags to flatc. Thus, flatc gets built targeting the target platform. To ensure flatc is built for the host, we can use include flatc as an ExternalProject. This does not propagate the CXX flags
• Some targets implicitly depended on flatc, we now make that requirement explicit
• We currently spread the FLATC_EXECUTABLE defaulting across the project. Let's just centralize this at the root

Test plan

$ ./install_executorch.sh
# Previously flatc was built against the Android target, now they target the host
$ ./build/build_android_library.sh
$ file /Users/jathu/executorch/cmake-out-android-arm64-v8a/third-party/flatbuffers/flatc
/Users/jathu/executorch/cmake-out-android-arm64-v8a/third-party/flatbuffers/flatc: Mach-O 64-bit executable arm64
# Apple builds work as usual, but they use flatc from pip
$ ./build/build_apple_frameworks.sh

cc @larryliu0820@lucylq@swolchok@dbort

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9077

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Cancelled Job, 2 Unrelated Failures

As of commit 4faa443 with merge base cb3ec19 ():

NEW FAILURE - The following job has failed:

• trunk / test-static-llama-ane / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

CANCELLED JOB - The following job was cancelled. Please retry:

• trunk / test-models-macos (cmake, linear, xnnpack-quantization-delegation, macos-m1-stable, 90) / macos-job (gh)
  ##[error]The operation was canceled.

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• trunk / test-arm-reference-delegation / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Apple / build-benchmark-app / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 65

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[swolchok]

looks great, just the one minor comment

[mergennachin]

@jathu I added 'ciflow/trunk' label, which triggers all trunk jobs

Looks like there are a bunch of CI failures

Please fix them.

[jathu]

These failing tests are unrelated to this diff:

• build-benchmark-app / macos-job — fatal error: 'omp.h' file not found is occurring in main
• test-arm-reference-delegation / linux-job — occurring in main
• test-models-macos (cmake, linear, xnnpack-quantization-delegation, macos-m1-stable, 90) / macos-job — similar omp.h problem as above
• test-static-llama-ane / macos-job — incorrect python version issue also occurring in main

[swolchok]

Now seeing this:

$ cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DEXECUTORCH_BUILD_KERNELS_CUSTOM=ON \
-DEXECUTORCH_BUILD_KERNELS_CUSTOM_AOT=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-DCMAKE_BUILD_TYPE=Release \
-DEXECUTORCH_ENABLE_LOGGING=ON \
-Bcmake-out -GNinja
$ cmake --build cmake-out
ninja: error: 'third-party/flatbuffers/flatc', needed by 'schema/include/executorch/schema/program_generated.h', missing and no known rule to make it

Repros on this commit and not the previous one.

[swolchok]

Ninja is extra-picky about dependencies. #9246 fixes

[jathu]

Summary

• FixesFlatc should be built for the host rather than the target when cross-compiling #7260
  • We use flatc as part of the build step to generate some files from FlatBuffer definitions. This implies the tool runs on the host. However, we currently build flatc as part of the main build — which propagates the target CXX flags to flatc. Thus, flatc gets built targeting the target platform. To ensure flatc is built for the host, we can use include flatc as an ExternalProject. This does not propagate the CXX flags
• Some targets implicitly depended on flatc, we now make that requirement explicit
• We currently spread the FLATC_EXECUTABLE defaulting across the project. Let's just centralize this at the root

Test plan

$ ./install_executorch.sh
# Previously flatc was built against the Android target, now they target the host
$ ./build/build_android_library.sh
$ file /Users/jathu/executorch/cmake-out-android-arm64-v8a/third-party/flatbuffers/flatc
/Users/jathu/executorch/cmake-out-android-arm64-v8a/third-party/flatbuffers/flatc: Mach-O 64-bit executable arm64
# Apple builds work as usual, but they use flatc from pip
$ ./build/build_apple_frameworks.sh

cc @larryliu0820@lucylq@swolchok@dbort

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9077

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Cancelled Job, 2 Unrelated Failures

As of commit 4faa443 with merge base cb3ec19 ():

NEW FAILURE - The following job has failed:

• trunk / test-static-llama-ane / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

CANCELLED JOB - The following job was cancelled. Please retry:

• trunk / test-models-macos (cmake, linear, xnnpack-quantization-delegation, macos-m1-stable, 90) / macos-job (gh)
  ##[error]The operation was canceled.

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• trunk / test-arm-reference-delegation / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Apple / build-benchmark-app / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 65

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[swolchok]

looks great, just the one minor comment

[mergennachin]

@jathu I added 'ciflow/trunk' label, which triggers all trunk jobs

Looks like there are a bunch of CI failures

Please fix them.

[jathu]

These failing tests are unrelated to this diff:

• build-benchmark-app / macos-job — fatal error: 'omp.h' file not found is occurring in main
• test-arm-reference-delegation / linux-job — occurring in main
• test-models-macos (cmake, linear, xnnpack-quantization-delegation, macos-m1-stable, 90) / macos-job — similar omp.h problem as above
• test-static-llama-ane / macos-job — incorrect python version issue also occurring in main

[swolchok]

Now seeing this:

$ cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DEXECUTORCH_BUILD_KERNELS_CUSTOM=ON \
-DEXECUTORCH_BUILD_KERNELS_CUSTOM_AOT=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-DCMAKE_BUILD_TYPE=Release \
-DEXECUTORCH_ENABLE_LOGGING=ON \
-Bcmake-out -GNinja
$ cmake --build cmake-out
ninja: error: 'third-party/flatbuffers/flatc', needed by 'schema/include/executorch/schema/program_generated.h', missing and no known rule to make it

Repros on this commit and not the previous one.

[swolchok]

Ninja is extra-picky about dependencies. #9246 fixes

[jathu]

Summary

• FixesFlatc should be built for the host rather than the target when cross-compiling #7260
  • We use flatc as part of the build step to generate some files from FlatBuffer definitions. This implies the tool runs on the host. However, we currently build flatc as part of the main build — which propagates the target CXX flags to flatc. Thus, flatc gets built targeting the target platform. To ensure flatc is built for the host, we can use include flatc as an ExternalProject. This does not propagate the CXX flags
• Some targets implicitly depended on flatc, we now make that requirement explicit
• We currently spread the FLATC_EXECUTABLE defaulting across the project. Let's just centralize this at the root

Test plan

$ ./install_executorch.sh
# Previously flatc was built against the Android target, now they target the host
$ ./build/build_android_library.sh
$ file /Users/jathu/executorch/cmake-out-android-arm64-v8a/third-party/flatbuffers/flatc
/Users/jathu/executorch/cmake-out-android-arm64-v8a/third-party/flatbuffers/flatc: Mach-O 64-bit executable arm64
# Apple builds work as usual, but they use flatc from pip
$ ./build/build_apple_frameworks.sh

cc @larryliu0820@lucylq@swolchok@dbort

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9077

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Cancelled Job, 2 Unrelated Failures

As of commit 4faa443 with merge base cb3ec19 ():

NEW FAILURE - The following job has failed:

• trunk / test-static-llama-ane / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

CANCELLED JOB - The following job was cancelled. Please retry:

• trunk / test-models-macos (cmake, linear, xnnpack-quantization-delegation, macos-m1-stable, 90) / macos-job (gh)
  ##[error]The operation was canceled.

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• trunk / test-arm-reference-delegation / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Apple / build-benchmark-app / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 65

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[swolchok]

looks great, just the one minor comment

[mergennachin]

@jathu I added 'ciflow/trunk' label, which triggers all trunk jobs

Looks like there are a bunch of CI failures

Please fix them.

[jathu]

These failing tests are unrelated to this diff:

• build-benchmark-app / macos-job — fatal error: 'omp.h' file not found is occurring in main
• test-arm-reference-delegation / linux-job — occurring in main
• test-models-macos (cmake, linear, xnnpack-quantization-delegation, macos-m1-stable, 90) / macos-job — similar omp.h problem as above
• test-static-llama-ane / macos-job — incorrect python version issue also occurring in main

[swolchok]

Now seeing this:

$ cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DEXECUTORCH_BUILD_KERNELS_CUSTOM=ON \
-DEXECUTORCH_BUILD_KERNELS_CUSTOM_AOT=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-DCMAKE_BUILD_TYPE=Release \
-DEXECUTORCH_ENABLE_LOGGING=ON \
-Bcmake-out -GNinja
$ cmake --build cmake-out
ninja: error: 'third-party/flatbuffers/flatc', needed by 'schema/include/executorch/schema/program_generated.h', missing and no known rule to make it

Repros on this commit and not the previous one.

[swolchok]

Ninja is extra-picky about dependencies. #9246 fixes

[jathu]

Summary

• FixesFlatc should be built for the host rather than the target when cross-compiling #7260
  • We use flatc as part of the build step to generate some files from FlatBuffer definitions. This implies the tool runs on the host. However, we currently build flatc as part of the main build — which propagates the target CXX flags to flatc. Thus, flatc gets built targeting the target platform. To ensure flatc is built for the host, we can use include flatc as an ExternalProject. This does not propagate the CXX flags
• Some targets implicitly depended on flatc, we now make that requirement explicit
• We currently spread the FLATC_EXECUTABLE defaulting across the project. Let's just centralize this at the root

Test plan

$ ./install_executorch.sh
# Previously flatc was built against the Android target, now they target the host
$ ./build/build_android_library.sh
$ file /Users/jathu/executorch/cmake-out-android-arm64-v8a/third-party/flatbuffers/flatc
/Users/jathu/executorch/cmake-out-android-arm64-v8a/third-party/flatbuffers/flatc: Mach-O 64-bit executable arm64
# Apple builds work as usual, but they use flatc from pip
$ ./build/build_apple_frameworks.sh

cc @larryliu0820@lucylq@swolchok@dbort

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/9077

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Cancelled Job, 2 Unrelated Failures

As of commit 4faa443 with merge base cb3ec19 ():

NEW FAILURE - The following job has failed:

• trunk / test-static-llama-ane / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

CANCELLED JOB - The following job was cancelled. Please retry:

• trunk / test-models-macos (cmake, linear, xnnpack-quantization-delegation, macos-m1-stable, 90) / macos-job (gh)
  ##[error]The operation was canceled.

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• trunk / test-arm-reference-delegation / linux-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but was present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Apple / build-benchmark-app / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 65

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[swolchok]

looks great, just the one minor comment

[mergennachin]

@jathu I added 'ciflow/trunk' label, which triggers all trunk jobs

Looks like there are a bunch of CI failures

Please fix them.

[jathu]

These failing tests are unrelated to this diff:

• build-benchmark-app / macos-job — fatal error: 'omp.h' file not found is occurring in main
• test-arm-reference-delegation / linux-job — occurring in main
• test-models-macos (cmake, linear, xnnpack-quantization-delegation, macos-m1-stable, 90) / macos-job — similar omp.h problem as above
• test-static-llama-ane / macos-job — incorrect python version issue also occurring in main

[swolchok]

Now seeing this:

$ cmake . \
-DCMAKE_INSTALL_PREFIX=cmake-out \
-DEXECUTORCH_BUILD_KERNELS_CUSTOM=ON \
-DEXECUTORCH_BUILD_KERNELS_CUSTOM_AOT=ON \
-DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON \
-DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \
-DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \
-DEXECUTORCH_BUILD_EXTENSION_MODULE=ON \
-DEXECUTORCH_BUILD_EXTENSION_RUNNER_UTIL=ON \
-DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \
-DEXECUTORCH_BUILD_EXTENSION_FLAT_TENSOR=ON \
-DEXECUTORCH_BUILD_DEVTOOLS=ON \
-DEXECUTORCH_BUILD_XNNPACK=ON \
-DEXECUTORCH_BUILD_TESTS=ON \
-DCMAKE_BUILD_TYPE=Release \
-DEXECUTORCH_ENABLE_LOGGING=ON \
-Bcmake-out -GNinja
$ cmake --build cmake-out
ninja: error: 'third-party/flatbuffers/flatc', needed by 'schema/include/executorch/schema/program_generated.h', missing and no known rule to make it

Repros on this commit and not the previous one.

[swolchok]

Ninja is extra-picky about dependencies. #9246 fixes