create and validate build_variables.bzl

[swolchok]

First step of #8268 -- we are moving from buck-extracted filelist to using one shared filelist, and the first step is to create the shared filelist and validate it against the buck generation.

Differential Revision: D80187441

[swolchok]

Stack from ghstack (oldest at bottom):

• switch top-level ExecuTorch build from executorch_srcs.cmake to build_variables.bzl #13358
• build_variables.bzl: make CUSTOM_OPS_SRCS usable in BUCK #13357
• build_variables.bzl: make XNNPACK_BACKEND_SRCS usable in BUCK #13356
• build_variables.bzl: make MPS_BACKEND_SRCS usable in BUCK #13355
• build_variables.bzl: make THREADPOOL_SRCS usable in BUCK #13354
• build_variables.bzl: split PATTERN_SRCS from PORTABLE_KERNELS_SRCS #13353
• build_variables.bzl: split PLATFORM_SRCS from EXECUTORCH_CORE_SRCS #8398
• build_variables.bzl: split PROGRAM_NO_PRIM_OPS_SRCS from EXECUTORCH_CORE_SRCS #8397
• ->create and validate build_variables.bzl #8326

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/8326

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit 7f2ca6a with merge base 70ab5ab ():

NEW FAILURE - The following job has failed:

• Build documentation / build (buck2) / Build doc (gh)
  At least one of the pre-conditions you specified did not hold

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[swolchok]

Current error:

Buck-generated _custom_ops__srcs does not match hardcoded CUSTOM_OPS_SRCS
in build_variables.bzl. Left:
extension/llm/custom_ops/op_fallback.cpp;extension/llm/custom_ops/op_fast_hadamard_transform.cpp;extension/llm/custom_ops/op_sdpa.cpp;extension/llm/custom_ops/op_update_cache.cpp;extension/llm/custom_ops/spinquant/fast_hadamard_transform.cpp;extension/llm/custom_ops/spinquant/third-party/FFHT/fht_avx.c;kernels/portable/cpu/util/reduce_util.cpp
Right: extension/llm/custom_ops/op_fallback.cpp;extension/llm/custom_ops/op_fast_hadamard_transform.cpp;extension/llm/custom_ops/op_sdpa.cpp;extension/llm/custom_ops/op_update_cache.cpp;extension/llm/custom_ops/spinquant/fast_hadamard_transform.cpp;extension/llm/custom_ops/spinquant/third-party/FFHT/fht_neon.c;kernels/portable/cpu/util/reduce_util.cpp

So, we need a mechanism for choosing fht_avx.c on Intel and fht_neon.c on ARM. Best way is probably to add a CMake-native way to do this and filter these files out of the Buck-to-CMake pipeline.

[swolchok]

@swolchok has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

[swolchok]

@swolchok has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.