ExecuTorch Build Problem

[emreaniloguz]

🐛 Describe the bug

Hello,

I'm attempting to follow the official ExecuTorch setup page, but when I execute the following command:

# Build the executor_runner target
cmake --build cmake-out --target executor_runner -j9

I encounter the following error:

Traceback (most recent call last):
 File "/home/emre/anaconda3/envs/executorch/lib/python3.10/runpy.py", line 196, in _run_module_as_main
 return _run_code(code, main_globals, None,
 File "/home/emre/anaconda3/envs/executorch/lib/python3.10/runpy.py", line 86, in _run_code
 exec(code, run_globals)
 File "/home/emre/anaconda3/envs/executorch/lib/python3.10/site-packages/torchgen/gen_executorch.py", line 994, in <module>
 main()
 File "/home/emre/anaconda3/envs/executorch/lib/python3.10/site-packages/torchgen/gen_executorch.py", line 932, in main
 parsed_yaml, custom_ops_parsed_yaml = parse_yaml_files(
 File "/home/emre/anaconda3/envs/executorch/lib/python3.10/site-packages/torchgen/gen_executorch.py", line 790, in parse_yaml_files
 translate_native_yaml(
 File "/home/emre/anaconda3/envs/executorch/lib/python3.10/site-packages/torchgen/gen_executorch.py", line 642, in translate_native_yaml
 native_functions, persisted_fields = parse_et_yaml(
 File "/home/emre/anaconda3/envs/executorch/lib/python3.10/site-packages/torchgen/executorch/parse.py", line 127, in parse_et_yaml
 with open(path) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/emre/anaconda3/envs/executorch/lib/python3.10/site-packages/torchgen/packaged/ATen/native/native_functions.yaml'
make[3]: *** [kernels/portable/CMakeFiles/portable_ops_lib.dir/build.make:142: kernels/portable/portable_ops_lib/RegisterCodegenUnboxedKernelsEverything.cpp] Error 1
make[2]: *** [CMakeFiles/Makefile2:506: kernels/portable/CMakeFiles/portable_ops_lib.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:300: CMakeFiles/executor_runner.dir/rule] Error 2
make: *** [Makefile:195: executor_runner] Error 2

I believe this issue is related to pytorch/pytorch#124941 because the install_requirements.sh script downloads the nightly version of PyTorch.

Versions

Collecting environment information...
PyTorch version: 2.4.0.dev20240422+cpu
Is debug build: False
CUDA used to build PyTorch: Could not collect
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: Could not collect
CMake version: version 3.29.2
Libc version: glibc-2.31

Python version: 3.10.14 (main, May 6 2024, 19:42:50) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-105-generic-x86_64-with-glibc2.31
Is CUDA available: False
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1650
Nvidia driver version: 535.171.04
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_precompiled.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_engines_runtime_compiled.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_graph.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_heuristic.so.9.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops.so.9.1.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 39 bits physical, 48 bits virtual
CPU(s): 12
On-line CPU(s) list: 0-11
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 165
Model name: Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz
Stepping: 2
CPU MHz: 2600.000
CPU max MHz: 5000,0000
CPU min MHz: 800,0000
BogoMIPS: 5199.98
Virtualization: VT-x
L1d cache: 192 KiB
L1i cache: 192 KiB
L2 cache: 1,5 MiB
L3 cache: 12 MiB
NUMA node0 CPU(s): 0-11
Vulnerability Gather data sampling: Mitigation; Microcode
Vulnerability Itlb multihit: KVM: Mitigation: VMX disabled
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Retbleed: Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds: Mitigation; Microcode
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp pku ospke md_clear flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] executorch==0.3.0a0+f8403ed
[pip3] numpy==1.26.4
[pip3] torch==2.4.0.dev20240422+cpu
[pip3] torchaudio==2.2.0.dev20240422+cpu
[pip3] torchsr==1.0.4
[pip3] torchvision==0.19.0.dev20240422+cpu
[pip3] triton==2.0.0
[conda] blas 1.0 mkl
[conda] executorch 0.3.0a0+f8403ed pypi_0 pypi
[conda] ffmpeg 4.3 hf484d3e_0 pytorch
[conda] mkl 2023.1.0 h213fc3f_46344
[conda] mkl-service 2.4.0 py310h5eee18b_1
[conda] mkl_fft 1.3.8 py310h5eee18b_0
[conda] mkl_random 1.2.4 py310hdb19cb5_0
[conda] numpy 1.26.4 py310h5f9d8c6_0
[conda] numpy-base 1.26.4 py310hb5e798b_0
[conda] pytorch-cuda 11.8 h7e8668a_5 pytorch
[conda] pytorch-mutex 1.0 cuda pytorch
[conda] torch 2.4.0.dev20240422+cpu pypi_0 pypi
[conda] torchaudio 2.2.0.dev20240422+cpu pypi_0 pypi
[conda] torchsr 1.0.4 pypi_0 pypi
[conda] torchtriton 2.0.0 py310 pytorch
[conda] torchvision 0.19.0.dev20240422+cpu pypi_0 pypi

[emreaniloguz]

Changing the NIGHTLY_VERSION=dev20240424 to NIGHTLY_VERSION=dev20240328 in install_requirements.sh worked in my case.

[malfet]

Move to ExecuTorch repo

[mergennachin]

@emreaniloguz thanks for the bug report.

I just updated the nightly version to 0507 (#3562)

Can you verify that it works on your platform? If it works, please close the issue.

[zingo]

Thanks for checking this, at least we still see this in our nightly CI when building this morning e.g. using:

HEAD is now at 749f6ab41 Rust: Update sysinfo crate (#3520)
 ...
07:29:19 [ 52%] Building CXX object kernels/portable/CMakeFiles/portable_kernels.dir/cpu/op_bitwise_and.cpp.o
07:29:19 [ 52%] Building CXX object kernels/portable/CMakeFiles/portable_kernels.dir/cpu/op_bitwise_not.cpp.o
07:29:19 [ 52%] Generating code for kernel registration
07:29:19 Traceback (most recent call last):
07:29:19 File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
07:29:20 [ 52%] Building CXX object kernels/portable/CMakeFiles/portable_kernels.dir/cpu/op_bitwise_or.cpp.o
07:29:20 return _run_code(code, main_globals, None,
07:29:20 File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
07:29:20 exec(code, run_globals)
07:29:20 File "/home/svc_mlsystest/.local/lib/python3.10/site-packages/torchgen/gen_executorch.py", line 994, in <module>
07:29:20 main()
07:29:20 File "/home/svc_mlsystest/.local/lib/python3.10/site-packages/torchgen/gen_executorch.py", line 932, in main
07:29:20 parsed_yaml, custom_ops_parsed_yaml = parse_yaml_files(
07:29:20 File "/home/svc_mlsystest/.local/lib/python3.10/site-packages/torchgen/gen_executorch.py", line 790, in parse_yaml_files
07:29:20 translate_native_yaml(
07:29:20 File "/home/svc_mlsystest/.local/lib/python3.10/site-packages/torchgen/gen_executorch.py", line 642, in translate_native_yaml
07:29:20 native_functions, persisted_fields = parse_et_yaml(
07:29:20 File "/home/svc_mlsystest/.local/lib/python3.10/site-packages/torchgen/executorch/parse.py", line 127, in parse_et_yaml
07:29:20 with open(path) as f:
07:29:20 FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib/python3/dist-packages/torchgen/packaged/ATen/native/native_functions.yaml'
07:29:20 gmake[3]: *** [kernels/quantized/CMakeFiles/quantized_ops_lib.dir/build.make:84: kernels/quantized/quantized_ops_lib/RegisterCodegenUnboxedKernelsEverything.cpp] Error 1
07:29:20 gmake[2]: *** [CMakeFiles/Makefile2:1014: kernels/quantized/CMakeFiles/quantized_ops_lib.dir/all] Error 2

[mergennachin]

@zingo

Thanks for checking this, at least we still see this in our nightly CI

Is there a link? Or is it internal to Arm? What is the torch version it is using?

Maybe it needs to run install_requirements.sh again. Or use the same version as .ci/docker/ci_commit_pins/pytorch.txt or NIGHTLY_VERSION in install_requirements.sh

[zingo]

It's unfortunate an internal job,

What is the torch version it is using?
Whatever version is pointer out from the project, I see this in the logs:

17:52:57 Installing collected packages: zstd, mpmath, typing-extensions, tqdm, sympy, safetensors, regex, pillow, numpy, networkx, MarkupSafe, fsspec, jinja2, huggingface-hub, torch, tokenizers, transformers, torchvision, torchaudio, torchsr, timm
17:53:23 Successfully installed MarkupSafe-2.1.5 fsspec-2024.3.1 huggingface-hub-0.23.0 jinja2-3.1.4 mpmath-1.3.0 networkx-3.3 numpy-1.26.4 pillow-10.3.0 regex-2024.5.10 safetensors-0.4.3 sympy-1.12 timm-0.6.13 tokenizers-0.15.2 torch-2.4.0.dev20240507+cpu torchaudio-2.2.0.dev20240507+cpu torchsr-1.0.4 torchvision-0.19.0.dev20240507+cpu tqdm-4.66.4 transformers-4.38.2 typing-extensions-4.11.0 zstd-1.5.5.1

The jobs start in a almost empty/fresh docker (ubuntu_version=22.04 e.g. with some basic non project tools installed like pip) and then basically just doing

pip install -U pip setuptools cmake pytest-cov
git clone <INTERNAL_MIRROR_URL> executorch
cd executorch
git submodule init
git submodule update --init
CMAKE_PREFIX_PATH=<INTERNAL_PATH>/.local/lib/python3.10/site-packages ./install_requirements.sh --pybind xnnpack

(Not using conda or venv in this test but we could add it if that is needed)

If we run it without --pybind xnnpack install_requirements.sh works. The reason we add this is that we wanted to run the unit tests and need it to pass some test.

Ill dig some more into this on Monday to see if I can find out some more! I could double check that we get a fresh docker and not something reused that we might have runned on 2 days back. The normal setup is that they are only reused if jobs are triggered back-back but this is some jenkins setting that someone might have played with.

Also maybe check if we can do this:

Or use the same version as .ci/docker/ci_commit_pins/pytorch.txt

instead of just calling install_requirements.sh (with it's NIGHTLY_VERSION )

I don't know if original reported of this bug still sees the problem.