[BUG] No supported gcc/g++ host compiler found. (torchCuda11.8 + transformer + deepspeed )

[flckv]

Environment setup error when loading cpuadam.

to reproduce:
I used DS_BUILD_OPS=0 pip install deepspeed because it was solution for similar issue: by @jeffra in #1846 (comment)_ and @stas00

  conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
  DS_BUILD_OPS=0 pip install deepspeed
  pip install git+https://github.com/huggingface/transformers

The compiler of the gpu is not accessed. So when I run the run_classification_w.py script, the output gives error that just loading the cpu adam with deepspeed has issues. See the highlighted parts:

1. run_classification_w.py:

import logging
import os
import sys
import warnings
from dataclasses import dataclass, field
from random import randint
from typing import Optional

import datasets
import evaluate
import numpy as np
from datasets import DatasetDict, load_dataset

import transformers
from transformers import (
AutoConfig,
AutoFeatureExtractor,
AutoModelForAudioClassification,
HfArgumentParser,
Trainer,
TrainingArguments,
set_seed,
)
from transformers.trainer_utils import get_last_checkpoint
from transformers.utils import check_min_version, send_example_telemetry
from transformers.utils.versions import require_version

logger = logging.getLogger(name)

from huggingface_hub import login
login(token="...")

import deepspeed
deepspeed.ops.op_builder.CPUAdamBuilder().load()

**error is raised at line: **: deepspeed.ops.op_builder.CPUAdamBuilder().load()

2. output:

…

[2023-07-06 17:55:36,866] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-07-06 17:55:37,457] [WARNING] [runner.py:196:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
Detected CUDA_VISIBLE_DEVICES=0: setting --include=localhost:0
[2023-07-06 17:55:39,112] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-07-06 17:55:39,673] [INFO] [launch.py:145:main] WORLD INFO DICT: {'localhost': [0]}
[2023-07-06 17:55:39,673] [INFO] [launch.py:151:main] nnodes=1, num_local_procs=1, node_rank=0
[2023-07-06 17:55:39,673] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0]})
[2023-07-06 17:55:39,673] [INFO] [launch.py:163:main] dist_world_size=1
[2023-07-06 17:55:39,673] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=0
Token will not been saved to git credential helper. Pass add_to_git_credential=True if you want to set the git credential as well.
Token is valid (permission: write).
[2023-07-06 17:55:44,585] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Using /home/flckv/.cache/torch_extensions/py311_cu118 as PyTorch extensions root...
Creating extension directory /home/flckv/.cache/torch_extensions/py311_cu118/cpu_adam...
Detected CUDA files, patching ldflags

Emitting ninja build file /home/flckv/.cache/torch_extensions/py311_cu118/cpu_adam/build.ninja...

…

Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/3] /usr/bin/nvcc -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -I/home/flckv/.conda/envs/w/lib/python3.11/site-packages/deepspeed/ops/csrc/includes -I/usr/include -isystem /home/flckv/.conda/envs/w/lib/python3.11/site-packages/torch/include -isystem /home/flckv/.conda/envs/w/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/flckv/.conda/envs/w/lib/python3.11/site-packages/torch/include/TH -isystem /home/flckv/.conda/envs/w/lib/python3.11/site-packages/torch/include/THC -isystem /home/flckv/.conda/envs/w/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_80,code=sm_80 --compiler-options '-fPIC' -O3 --use_fast_math -std=c++17 -U__CUDA_NO_HALF_OPERATORS_ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_80,code=compute_80 -DBF16_AVAILABLE -c /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/deepspeed/ops/csrc/common/custom_cuda_kernel.cu -o custom_cuda_kernel.cuda.o

FAILED: custom_cuda_kernel.cuda.o

…

/usr/bin/nvcc -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1011" -I/home/flckv/.conda/envs/w/lib/python3.11/site-packages/deepspeed/ops/csrc/includes -I/usr/include -isystem /home/flckv/.conda/envs/w/lib/python3.11/site-packages/torch/include -isystem /home/flckv/.conda/envs/w/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/flckv/.conda/envs/w/lib/python3.11/site-packages/torch/include/TH -isystem /home/flckv/.conda/envs/w/lib/python3.11/site-packages/torch/include/THC -isystem /home/flckv/.conda/envs/w/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS -D__CUDA_NO_BFLOAT16_CONVERSIONS -D__CUDA_NO_HALF2_OPERATORS --expt-relaxed-constexpr -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_80,code=sm_80 --compiler-options '-fPIC' -O3 --use_fast_math -std=c++17 -U__CUDA_NO_HALF_OPERATORS_ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_80,code=compute_80 -DBF16_AVAILABLE -c /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/deepspeed/ops/csrc/common/custom_cuda_kernel.cu -o custom_cuda_kernel.cuda.o

 ERROR: No supported gcc/g++ host compiler found.
        Use 'nvcc -ccbin <compiler>' to specify a host compiler.

…

[2/3] c++ -MMD -MF cpu_adam.o.d -DTORCH_EXTENSION_NAME=cpu_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="_gcc" -DPYBIND11_STDLIB="_libstdcpp" -DPYBIND11_BUILD_ABI="_cxxabi1011" -I/home/flckv/.conda/envs/_w/lib/python3.11/site-packages/deepspeed/ops/csrc/includes -I/usr/include -isystem /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/torch/include -isystem /home/flckv/.conda/envs/w/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -isystem /home/flckv/.conda/envs/w/lib/python3.11/site-packages/torch/include/TH -isystem /home/flckv/.conda/envs/w/lib/python3.11/site-packages/torch/include/THC -isystem /home/flckv/.conda/envs/w/include/python3.11 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++17 -g -Wno-reorder -L/usr/lib64 -lcudart -lcublas -g -march=native -fopenmp -D__AVX256 -D__ENABLE_CUDA -DBF16_AVAILABLE -c /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/deepspeed/ops/csrc/adam/cpu_adam.cpp -o cpu_adam.o
ninja: build stopped: subcommand failed.

cpp_extension.py error:

Traceback (most recent call last):
File "/home/flckv/.conda/envs/_w/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1893, in _run_ninja_build
subprocess.run(
File "/home/flckv/.conda/envs/_w/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/flckv/transformers/examples/pytorch/audio-classification/run_audio_classification_w.py", line 51, in
deepspeed.ops.op_builder.CPUAdamBuilder().load()
File "/home/flckv/.conda/envs/_w/lib/python3.11/site-packages/deepspeed/ops/op_builder/builder.py", line 454, in load
return self.jit_load(verbose)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/flckv/.conda/envs/_w/lib/python3.11/site-packages/deepspeed/ops/op_builder/builder.py", line 497, in jit_load
op_module = load(name=self.name,
^^^^^^^^^^^^^^^^^^^^
File "/home/flckv/.conda/envs/_w/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1284, in load
return _jit_compile(
^^^^^^^^^^^^^
File "/home/flckv/.conda/envs/_w/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1509, in _jit_compile
_write_ninja_file_and_build_library(
File "/home/flckv/.conda/envs/_w/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1624, in _write_ninja_file_and_build_library
_run_ninja_build(
File "/home/flckv/.conda/envs/_w/lib/python3.11/site-packages/torch/utils/cpp_extension.py", line 1909, in _run_ninja_build
raise RuntimeError(message) from e

RuntimeError: Error building extension 'cpu_adam'

• transformers version: 4.31.0.dev0
• Platform: Linux-5.4.204-ql-generic-12.0-19-x86_64-with-glibc2.31
• Python version: 3.11.4
• Huggingface_hub version: 0.16.2
• Safetensors version: 0.3.1
• PyTorch version (GPU?): 2.0.1 (True)

# DEEPSPEED DS_REPORT: 
[2023-07-06 15:42:37,686] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. �[92m[OKAY]�[0m
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
�[93m [WARNING] �[0m async_io requires the dev libaio .so object and headers but these were not found.
�[93m [WARNING] �[0m async_io: please install the libaio-dev package with apt
�[93m [WARNING] �[0m If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... �[93m[NO]�[0m ....... �[93m[NO]�[0m
cpu_adagrad ............ �[93m[NO]�[0m ....... �[92m[OKAY]�[0m
cpu_adam ............... �[93m[NO]�[0m ....... �[92m[OKAY]�[0m
fused_adam ............. �[93m[NO]�[0m ....... �[92m[OKAY]�[0m
fused_lamb ............. �[93m[NO]�[0m ....... �[92m[OKAY]�[0m
quantizer .............. �[93m[NO]�[0m ....... �[92m[OKAY]�[0m
random_ltd ............. �[93m[NO]�[0m ....... �[92m[OKAY]�[0m
�[93m [WARNING] �[0m sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.0
�[93m [WARNING] �[0m using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ �[93m[NO]�[0m ....... �[93m[NO]�[0m
spatial_inference ...... �[93m[NO]�[0m ....... �[92m[OKAY]�[0m
transformer ............ �[93m[NO]�[0m ....... �[92m[OKAY]�[0m
stochastic_transformer . �[93m[NO]�[0m ....... �[92m[OKAY]�[0m
transformer_inference .. �[93m[NO]�[0m ....... �[92m[OKAY]�[0m
--------------------------------------------------

DeepSpeed general environment info:
torch install path ............... ['/home/flck/.conda/envs/w_/lib/python3.11/site-packages/torch']
torch version .................... 2.0.1
deepspeed install path ........... ['/home/flck/.conda/envs/w_/lib/python3.11/site-packages/deepspeed']
deepspeed info ................... 0.10.0+cc3a7c9c, cc3a7c9c, master
torch cuda version ............... 11.8
torch hip version ................ None
nvcc version ..................... 11.8
deepspeed wheel compiled w. ...... torch 2.0, cuda 11.8





 
# NVCC and GCC:
  
**nvcc**: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0
Using built-in specs.

**COLLECT_GCC**=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/9/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:hsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu

> Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.4.0-1ubuntu1~20.04.1' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-9-Av3uEd/gcc-9-9.4.0/debian/tmp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
> Thread model: posix

gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)

[flckv]

I repeated all the above steps except I installed deepspeed with:
DS_BUILD_CPU_ADAM=1 pip install deepspeed

error while installing deepspeed:

…

copying deepspeed/runtime/pipe/p2p.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/pipe
copying deepspeed/runtime/pipe/schedule.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/pipe
copying deepspeed/runtime/pipe/topology.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/pipe
creating build/lib.linux-x86_64-cpython-311/deepspeed/runtime/swap_tensor
copying deepspeed/runtime/swap_tensor/init.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/swap_tensor
copying deepspeed/runtime/swap_tensor/aio_config.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/swap_tensor
copying deepspeed/runtime/swap_tensor/async_swapper.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/swap_tensor
copying deepspeed/runtime/swap_tensor/constants.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/swap_tensor
copying deepspeed/runtime/swap_tensor/optimizer_utils.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/swap_tensor
copying deepspeed/runtime/swap_tensor/partitioned_optimizer_swapper.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/swap_tensor
copying deepspeed/runtime/swap_tensor/partitioned_param_swapper.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/swap_tensor
copying deepspeed/runtime/swap_tensor/pipelined_optimizer_swapper.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/swap_tensor
copying deepspeed/runtime/swap_tensor/utils.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/swap_tensor
creating build/lib.linux-x86_64-cpython-311/deepspeed/runtime/zero
copying deepspeed/runtime/zero/init.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/zero
copying deepspeed/runtime/zero/config.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/zero
copying deepspeed/runtime/zero/contiguous_memory_allocator.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/zero
copying deepspeed/runtime/zero/linear.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/zero
copying deepspeed/runtime/zero/mics.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/zero
copying deepspeed/runtime/zero/mics_utils.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/zero
copying deepspeed/runtime/zero/offload_config.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/zero
copying deepspeed/runtime/zero/parameter_offload.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/zero
copying deepspeed/runtime/zero/partition_parameters.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/zero
copying deepspeed/runtime/zero/partitioned_param_coordinator.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/zero
copying deepspeed/runtime/zero/stage3.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/zero
copying deepspeed/runtime/zero/stage_1_and_2.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/zero
copying deepspeed/runtime/zero/test.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/zero
copying deepspeed/runtime/zero/tiling.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/zero
copying deepspeed/runtime/zero/utils.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/zero
creating build/lib.linux-x86_64-cpython-311/deepspeed/runtime/data_pipeline/data_routing
copying deepspeed/runtime/data_pipeline/data_routing/init.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/data_pipeline/data_routing
copying deepspeed/runtime/data_pipeline/data_routing/basic_layer.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/data_pipeline/data_routing
copying deepspeed/runtime/data_pipeline/data_routing/helper.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/data_pipeline/data_routing
copying deepspeed/runtime/data_pipeline/data_routing/scheduler.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/data_pipeline/data_routing
copying deepspeed/runtime/data_pipeline/data_routing/utils.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/data_pipeline/data_routing
creating build/lib.linux-x86_64-cpython-311/deepspeed/runtime/data_pipeline/data_sampling
copying deepspeed/runtime/data_pipeline/data_sampling/init.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/data_pipeline/data_sampling
copying deepspeed/runtime/data_pipeline/data_sampling/data_analyzer.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/data_pipeline/data_sampling
copying deepspeed/runtime/data_pipeline/data_sampling/data_sampler.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/data_pipeline/data_sampling
copying deepspeed/runtime/data_pipeline/data_sampling/indexed_dataset.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/data_pipeline/data_sampling
copying deepspeed/runtime/data_pipeline/data_sampling/utils.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/data_pipeline/data_sampling
creating build/lib.linux-x86_64-cpython-311/deepspeed/runtime/fp16/onebit
copying deepspeed/runtime/fp16/onebit/init.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/fp16/onebit
copying deepspeed/runtime/fp16/onebit/adam.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/fp16/onebit
copying deepspeed/runtime/fp16/onebit/lamb.py -> build/lib.linux-x86_64-cpython-311/deepspeed/runtime/fp16/onebit
copying deepspeed/runtime/fp16/onebit/zoadam.py -> build/lib.linux-x86_64-cpython-

311/deepspeed/runtime/fp16/onebit:

311/deepspeed/runtime/fp16/onebit
running egg_info
writing deepspeed.egg-info/PKG-INFO
writing dependency_links to deepspeed.egg-info/dependency_links.txt
writing entry points to deepspeed.egg-info/entry_points.txt
writing requirements to deepspeed.egg-info/requires.txt
writing top-level names to deepspeed.egg-info/top_level.txt
reading manifest file 'deepspeed.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '.hip' under directory 'deepspeed'
warning: no files found matching '.cc' under directory 'deepspeed'
warning: no files found matching '.tr' under directory 'csrc'
warning: no files found matching '.cc' under directory 'csrc'
warning: no files found matching '*.py' under directory 'benchmarks'
adding license file 'LICENSE'
writing manifest file 'deepspeed.egg-info/SOURCES.txt'
/home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.autotuning.config_templates' is absent from the packages configuration.

  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          **Python recognizes 'deepspeed.autotuning.config_templates' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.**

…

          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.autotuning.config_templates' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.autotuning.config_templates' to be distributed and are
          already explicitly excluding 'deepspeed.autotuning.config_templates' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html

          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.ops.csrc.adagrad' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'deepspeed.ops.csrc.adagrad' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.

…

          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.ops.csrc.adagrad' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.ops.csrc.adagrad' to be distributed and are
          already explicitly excluding 'deepspeed.ops.csrc.adagrad' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.ops.csrc.adam' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'deepspeed.ops.csrc.adam' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.ops.csrc.adam' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.ops.csrc.adam' to be distributed and are
          already explicitly excluding 'deepspeed.ops.csrc.adam' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.ops.csrc.aio.common' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'deepspeed.ops.csrc.aio.common' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.ops.csrc.aio.common' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.ops.csrc.aio.common' to be distributed and are
          already explicitly excluding 'deepspeed.ops.csrc.aio.common' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.ops.csrc.aio.py_lib' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'deepspeed.ops.csrc.aio.py_lib' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.ops.csrc.aio.py_lib' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.ops.csrc.aio.py_lib' to be distributed and are
          already explicitly excluding 'deepspeed.ops.csrc.aio.py_lib' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.ops.csrc.aio.py_test' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'deepspeed.ops.csrc.aio.py_test' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.ops.csrc.aio.py_test' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.ops.csrc.aio.py_test' to be distributed and are
          already explicitly excluding 'deepspeed.ops.csrc.aio.py_test' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.ops.csrc.common' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'deepspeed.ops.csrc.common' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.ops.csrc.common' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.ops.csrc.common' to be distributed and are
          already explicitly excluding 'deepspeed.ops.csrc.common' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.ops.csrc.cpu.comm' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'deepspeed.ops.csrc.cpu.comm' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.ops.csrc.cpu.comm' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.ops.csrc.cpu.comm' to be distributed and are
          already explicitly excluding 'deepspeed.ops.csrc.cpu.comm' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.ops.csrc.includes' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'deepspeed.ops.csrc.includes' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.ops.csrc.includes' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.ops.csrc.includes' to be distributed and are
          already explicitly excluding 'deepspeed.ops.csrc.includes' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.ops.csrc.lamb' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'deepspeed.ops.csrc.lamb' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.ops.csrc.lamb' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.ops.csrc.lamb' to be distributed and are
          already explicitly excluding 'deepspeed.ops.csrc.lamb' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.ops.csrc.quantization' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'deepspeed.ops.csrc.quantization' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.ops.csrc.quantization' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.ops.csrc.quantization' to be distributed and are
          already explicitly excluding 'deepspeed.ops.csrc.quantization' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.ops.csrc.random_ltd' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'deepspeed.ops.csrc.random_ltd' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.ops.csrc.random_ltd' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.ops.csrc.random_ltd' to be distributed and are
          already explicitly excluding 'deepspeed.ops.csrc.random_ltd' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.ops.csrc.sparse_attention' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'deepspeed.ops.csrc.sparse_attention' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.ops.csrc.sparse_attention' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.ops.csrc.sparse_attention' to be distributed and are
          already explicitly excluding 'deepspeed.ops.csrc.sparse_attention' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.ops.csrc.spatial.csrc' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'deepspeed.ops.csrc.spatial.csrc' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.ops.csrc.spatial.csrc' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.ops.csrc.spatial.csrc' to be distributed and are
          already explicitly excluding 'deepspeed.ops.csrc.spatial.csrc' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.ops.csrc.spatial.includes' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'deepspeed.ops.csrc.spatial.includes' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.ops.csrc.spatial.includes' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.ops.csrc.spatial.includes' to be distributed and are
          already explicitly excluding 'deepspeed.ops.csrc.spatial.includes' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.ops.csrc.transformer' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'deepspeed.ops.csrc.transformer' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.ops.csrc.transformer' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.ops.csrc.transformer' to be distributed and are
          already explicitly excluding 'deepspeed.ops.csrc.transformer' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.ops.csrc.transformer.inference.csrc' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'deepspeed.ops.csrc.transformer.inference.csrc' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.ops.csrc.transformer.inference.csrc' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.ops.csrc.transformer.inference.csrc' to be distributed and are
          already explicitly excluding 'deepspeed.ops.csrc.transformer.inference.csrc' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.ops.csrc.transformer.inference.includes' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'deepspeed.ops.csrc.transformer.inference.includes' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.ops.csrc.transformer.inference.includes' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.ops.csrc.transformer.inference.includes' to be distributed and are
          already explicitly excluding 'deepspeed.ops.csrc.transformer.inference.includes' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  
  
          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************
  
  !!
    check.warn(importable)
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/setuptools/command/build_py.py:201: _Warning: Package 'deepspeed.ops.csrc.utils' is absent from the `packages` configuration.
  !!
  
          ********************************************************************************
          ############################
          # Package would be ignored #
          ############################
          Python recognizes 'deepspeed.ops.csrc.utils' as an importable package[^1],
          but it is absent from setuptools' `packages` configuration.
  
          This leads to an ambiguous overall configuration. If you want to distribute this
          package, please make sure that 'deepspeed.ops.csrc.utils' is explicitly added
          to the `packages` configuration field.
  
          Alternatively, you can also rely on setuptools' discovery methods
          (for example by using `find_namespace_packages(...)`/`find_namespace:`
          instead of `find_packages(...)`/`find:`).
  
          You can read more about "package discovery" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  
          If you don't want 'deepspeed.ops.csrc.utils' to be distributed and are
          already explicitly excluding 'deepspeed.ops.csrc.utils' via
          `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
          you can try to use `exclude_package_data`, or `include-package-data=False` in
          combination with a more fine grained `package-data` configuration.
  
          You can read more about "package data files" on setuptools documentation page:
  
          - https://setuptools.pypa.io/en/latest/userguide/datafiles.html

          [^1]: For Python, any directory (with suitable naming) can be imported,
                even if it does not contain any `.py` files.
                On the other hand, currently there is no concept of package data
                directory, all directories are treated like packages.
          ********************************************************************************

check.warn(importable) creating build/lib.linux-x86_64-cpython-311/deepspeed/autotuning/config_templates

  !!
    check.warn(importable)
  creating build/lib.linux-x86_64-cpython-311/deepspeed/autotuning/config_templates
  copying deepspeed/autotuning/config_templates/template_zero0.json -> build/lib.linux-x86_64-cpython-311/deepspeed/autotuning/config_templates
  copying deepspeed/autotuning/config_templates/template_zero1.json -> build/lib.linux-x86_64-cpython-311/deepspeed/autotuning/config_templates
  copying deepspeed/autotuning/config_templates/template_zero2.json -> build/lib.linux-x86_64-cpython-311/deepspeed/autotuning/config_templates
  copying deepspeed/autotuning/config_templates/template_zero3.json -> build/lib.linux-x86_64-cpython-311/deepspeed/autotuning/config_templates
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/adagrad
  copying deepspeed/ops/csrc/adagrad/cpu_adagrad.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/adagrad
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/adam
  copying deepspeed/ops/csrc/adam/cpu_adam.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/adam
  copying deepspeed/ops/csrc/adam/fused_adam_frontend.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/adam
  copying deepspeed/ops/csrc/adam/multi_tensor_adam.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/adam
  copying deepspeed/ops/csrc/adam/multi_tensor_apply.cuh -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/adam
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/common
  copying deepspeed/ops/csrc/aio/common/deepspeed_aio_common.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/common
  copying deepspeed/ops/csrc/aio/common/deepspeed_aio_common.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/common
  copying deepspeed/ops/csrc/aio/common/deepspeed_aio_types.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/common
  copying deepspeed/ops/csrc/aio/common/deepspeed_aio_types.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/common
  copying deepspeed/ops/csrc/aio/common/deepspeed_aio_utils.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/common
  copying deepspeed/ops/csrc/aio/common/deepspeed_aio_utils.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/common
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/py_lib
  copying deepspeed/ops/csrc/aio/py_lib/deepspeed_aio_thread.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/py_lib
  copying deepspeed/ops/csrc/aio/py_lib/deepspeed_aio_thread.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/py_lib
  copying deepspeed/ops/csrc/aio/py_lib/deepspeed_pin_tensor.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/py_lib
  copying deepspeed/ops/csrc/aio/py_lib/deepspeed_pin_tensor.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/py_lib
  copying deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/py_lib
  copying deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/py_lib
  copying deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio_handle.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/py_lib
  copying deepspeed/ops/csrc/aio/py_lib/deepspeed_py_aio_handle.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/py_lib
  copying deepspeed/ops/csrc/aio/py_lib/deepspeed_py_copy.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/py_lib
  copying deepspeed/ops/csrc/aio/py_lib/deepspeed_py_copy.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/py_lib
  copying deepspeed/ops/csrc/aio/py_lib/py_ds_aio.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/py_lib
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/py_test
  copying deepspeed/ops/csrc/aio/py_test/single_process_config.json -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/aio/py_test
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/common
  copying deepspeed/ops/csrc/common/custom_cuda_kernel.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/common
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/cpu
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/cpu/comm
  copying deepspeed/ops/csrc/cpu/comm/ccl.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/cpu/comm
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/StopWatch.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/Timer.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/compat.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/context.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/conversion_utils.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/cpu_adagrad.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/cpu_adam.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/cublas_wrappers.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/custom_cuda_layers.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/dequantization_utils.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/dropout.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/ds_kernel_utils.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/ds_transformer_cuda.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/feed_forward.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/gelu.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/gemm_test.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/general_kernels.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/memory_access_utils.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/normalize_layer.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/quantization.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/quantization_utils.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/quantizer.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/reduction_utils.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/simd.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/softmax.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/strided_batch_gemm.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  copying deepspeed/ops/csrc/includes/type_shim.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/includes
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/lamb
  copying deepspeed/ops/csrc/lamb/fused_lamb_cuda.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/lamb
  copying deepspeed/ops/csrc/lamb/fused_lamb_cuda_kernel.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/lamb
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/quantization
  copying deepspeed/ops/csrc/quantization/dequantize.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/quantization
  copying deepspeed/ops/csrc/quantization/fake_quantizer.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/quantization
  copying deepspeed/ops/csrc/quantization/pt_binding.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/quantization
  copying deepspeed/ops/csrc/quantization/quantize.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/quantization
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/random_ltd
  copying deepspeed/ops/csrc/random_ltd/gather_scatter.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/random_ltd
  copying deepspeed/ops/csrc/random_ltd/pt_binding.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/random_ltd
  copying deepspeed/ops/csrc/random_ltd/slice_attn_masks.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/random_ltd
  copying deepspeed/ops/csrc/random_ltd/token_sort.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/random_ltd
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/sparse_attention
  copying deepspeed/ops/csrc/sparse_attention/utils.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/sparse_attention
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/spatial
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/spatial/csrc
  copying deepspeed/ops/csrc/spatial/csrc/opt_bias_add.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/spatial/csrc
  copying deepspeed/ops/csrc/spatial/csrc/pt_binding.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/spatial/csrc
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/spatial/includes
  copying deepspeed/ops/csrc/spatial/includes/spatial_cuda_layers.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/spatial/includes
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer
  copying deepspeed/ops/csrc/transformer/cublas_wrappers.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer
  copying deepspeed/ops/csrc/transformer/dropout_kernels.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer
  copying deepspeed/ops/csrc/transformer/ds_transformer_cuda.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer
  copying deepspeed/ops/csrc/transformer/gelu_kernels.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer
  copying deepspeed/ops/csrc/transformer/general_kernels.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer
  copying deepspeed/ops/csrc/transformer/normalize_kernels.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer
  copying deepspeed/ops/csrc/transformer/softmax_kernels.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer
  copying deepspeed/ops/csrc/transformer/transform_kernels.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer

creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer/inference

  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer/inference
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer/inference/csrc
  copying deepspeed/ops/csrc/transformer/inference/csrc/apply_rotary_pos_emb.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer/inference/csrc
  copying deepspeed/ops/csrc/transformer/inference/csrc/dequantize.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer/inference/csrc
  copying deepspeed/ops/csrc/transformer/inference/csrc/gelu.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer/inference/csrc
  copying deepspeed/ops/csrc/transformer/inference/csrc/layer_norm.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer/inference/csrc
  copying deepspeed/ops/csrc/transformer/inference/csrc/pointwise_ops.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer/inference/csrc
  copying deepspeed/ops/csrc/transformer/inference/csrc/pt_binding.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer/inference/csrc
  copying deepspeed/ops/csrc/transformer/inference/csrc/relu.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer/inference/csrc
  copying deepspeed/ops/csrc/transformer/inference/csrc/rms_norm.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer/inference/csrc
  copying deepspeed/ops/csrc/transformer/inference/csrc/softmax.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer/inference/csrc
  copying deepspeed/ops/csrc/transformer/inference/csrc/transform.cu -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer/inference/csrc
  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer/inference/includes
  copying deepspeed/ops/csrc/transformer/inference/includes/inference_context.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer/inference/includes
  copying deepspeed/ops/csrc/transformer/inference/includes/inference_cublas_wrappers.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer/inference/includes
  copying deepspeed/ops/csrc/transformer/inference/includes/inference_cuda_layers.h -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/transformer/inference/includes

  creating build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/utils
  copying deepspeed/ops/csrc/utils/flatten_unflatten.cpp -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/csrc/utils
  copying deepspeed/ops/sparse_attention/trsrc/matmul.tr -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/sparse_attention/trsrc
  copying deepspeed/ops/sparse_attention/trsrc/softmax_bwd.tr -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/sparse_attention/trsrc
  copying deepspeed/ops/sparse_attention/trsrc/softmax_fwd.tr -> build/lib.linux-x86_64-cpython-311/deepspeed/ops/sparse_attention/trsrc
  running build_ext
  /home/flckv/.conda/envs/_w/lib/python3.11/site-packages/torch/utils/cpp_extension.py:398: UserWarning: There are no g++ version bounds defined for CUDA version 11.8
    warnings.warn(f'There are no {compiler_name} version bounds defined for CUDA version {cuda_str_version}')
  building 'deepspeed.ops.adam.cpu_adam_op' extension
  creating build/temp.linux-x86_64-cpython-311
  creating build/temp.linux-x86_64-cpython-311/csrc
  creating build/temp.linux-x86_64-cpython-311/csrc/adam
  creating build/temp.linux-x86_64-cpython-311/csrc/common
  gcc -pthread -B /home/flckv/.conda/envs/_w/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /home/flckv/.conda/envs/_w/include -fPIC -O2 -isystem /home/flckv/.conda/envs/_w/include -fPIC -Icsrc/includes -I/usr/include -I/home/flckv/.conda/envs/_w/lib/python3.11/site-packages/torch/include -I/home/flckv/.conda/envs/_w/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/flckv/.conda/envs/_w/lib/python3.11/site-packages/torch/include/TH -I/home/flckv/.conda/envs/_w/lib/python3.11/site-packages/torch/include/THC -I/home/flckv/.conda/envs/_w/include/python3.11 -c csrc/adam/cpu_adam.cpp -o build/temp.linux-x86_64-cpython-311/csrc/adam/cpu_adam.o -O3 -std=c++17 -g -Wno-reorder -L/usr/lib64 -lcudart -lcublas -g -march=native -fopenmp -D__AVX256__ -D__ENABLE_CUDA__ -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=cpu_adam_op -D_GLIBCXX_USE_CXX11_ABI=0
  In file included from csrc/includes/cpu_adam.h:13,
                   from csrc/adam/cpu_adam.cpp:6:

…

  csrc/includes/simd.h:69: warning: ignoring #pragma unroll  [-Wunknown-pragmas]
     69 | #pragma unroll
        |
  csrc/includes/simd.h:76: warning: ignoring #pragma unroll  [-Wunknown-pragmas]
     76 | #pragma unroll
        |
  csrc/includes/simd.h:82: warning: ignoring #pragma unroll  [-Wunknown-pragmas]
     82 | #pragma unroll
        |
  csrc/includes/simd.h:90: warning: ignoring #pragma unroll  [-Wunknown-pragmas]
     90 | #pragma unroll
        |
  csrc/includes/simd.h:98: warning: ignoring #pragma unroll  [-Wunknown-pragmas]
     98 | #pragma unroll
        |
  csrc/includes/simd.h:106: warning: ignoring #pragma unroll  [-Wunknown-pragmas]
    106 | #pragma unroll
        |
  csrc/includes/simd.h:112: warning: ignoring #pragma unroll  [-Wunknown-pragmas]
    112 | #pragma unroll
        |
  csrc/includes/simd.h:118: warning: ignoring #pragma unroll  [-Wunknown-pragmas]
    118 | #pragma unroll
        |
  csrc/includes/simd.h:124: warning: ignoring #pragma unroll  [-Wunknown-pragmas]
    124 | #pragma unroll
        |
  csrc/includes/simd.h:130: warning: ignoring #pragma unroll  [-Wunknown-pragmas]
    130 | #pragma unroll
        |
  csrc/includes/simd.h:136: warning: ignoring #pragma unroll  [-Wunknown-pragmas]
    136 | #pragma unroll
        |

  csrc/adam/cpu_adam.cpp: In member function ‘void Adam_Optimizer::Step_1(float*, float*, float*, float*, size_t, ds_half_precision_t*, bool)’:
  csrc/adam/cpu_adam.cpp:52:30: warning: ‘params_cast_h’ may be used uninitialized in this function [-Wmaybe-uninitialized]
     52 |         ds_half_precision_t* params_cast_h;
        |                              ^~~~~~~~~~~~~
  csrc/adam/cpu_adam.cpp:51:30: warning: ‘grads_cast_h’ may be used uninitialized in this function [-Wmaybe-uninitialized]
     51 |         ds_half_precision_t* grads_cast_h;
        |                              ^~~~~~~~~~~~
  /usr/bin/nvcc -Icsrc/includes -I/usr/include -I/home/flckv/.conda/envs/_w/lib/python3.11/site-packages/torch/include -I/home/flckv/.conda/envs/_w/lib/python3.11/site-packages/torch/include/torch/csrc/api/include -I/home/flckv/.conda/envs/_w/lib/python3.11/site-packages/torch/include/TH -I/home/flckv/.conda/envs/_w/lib/python3.11/site-packages/torch/include/THC -I/home/flckv/.conda/envs/_w/include/python3.11 -c csrc/common/custom_cuda_kernel.cu -o build/temp.linux-x86_64-cpython-311/csrc/common/custom_cuda_kernel.o -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr --compiler-options '-fPIC' -O3 --use_fast_math -std=c++17 -U__CUDA_NO_HALF_OPERATORS__ -U__CUDA_NO_HALF_CONVERSIONS__ -U__CUDA_NO_HALF2_OPERATORS__ -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=sm_86 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -DTORCH_EXTENSION_NAME=cpu_adam_op -D_GLIBCXX_USE_CXX11_ABI=0
  ERROR: No supported gcc/g++ host compiler found.
         Use 'nvcc -ccbin <compiler>' to specify a host compiler.
  error: command '/usr/bin/nvcc' failed with exit code 1
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for deepspeed
Running setup.py clean for deepspeed
Failed to build deepspeed
ERROR: Could not build wheels for deepspeed, which is required to install pyproject.toml-based projects

[CHNAICORP]

这是来自QQ邮箱的假期自动回复邮件。你好，我最近正在休假中，无法亲自回复你的邮件。我将在假期结束后，尽快给你回复。

[flckv]

I also tried installing deepspeed from source with these packages:

conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

CUDA_VISIBLE_DEVICES=0 python -c "import torch; print(torch.cuda.get_device_capability())":
(8, 0)

python -c "import torch; print(torch.cuda.get_arch_list())":
['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'sm_90', 'compute_37']

CUDA_VISIBLE_DEVICES=0 python -c "import torch; print(torch.cuda.get_device_properties(torch.device(cuda)))"
_CudaDeviceProperties(name='NVIDIA A100-SXM4-40GB', major=8, minor=0, total_memory=40536MB, multi_processor_count=108)

git clone https://github.com/microsoft/DeepSpeed/
cd DeepSpeed
rm -rf build
TORCH_CUDA_ARCH_LIST=“8.0” DS_BUILD_CPU_ADAM=1 DS_BUILD_UTILS=1; python setup.py build_ext -j8 bdist_wheel
pip install .

pip install -r ../requirements.txt

cd transformers
pip install .

but same error

[loadams]

Hi @flckv - is your torch installation built for the version of cuda you are using? Also what GPU are you using?

[flckv]

hi @loadams version of cuda: 11.6; GPU: NVIDIA A100-SXM4-40GB

So based on these I selected from https://pytorch.org/get-started/locally/: Stable(2.0.1), Linux, Conda, Python, Cuda 11.8: conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

[flckv]

@loadams any idea why deepspeed is not working?

[loadams]

Hi @flckv - can you clarify a few things for me?

If you run with DS_BUILD_OPS=0 pip install deepspeed that you don't see any errors in installation, but that's because no ops/cuda/nvcc calls are being made. However, if you then try to run and JIT compile things you hit errors? And those are the same errors you hit if you run DS_BUILD_OPS=1 pip install deepspeed?

And that the first error that you hit in both cases is ERROR: No supported gcc/g++ host compiler found.?

Could you please also include the output of the following?

python -c "import torch; print('torch:', torch.__version__, torch)"
python -c "import torch; print('CUDA available:', torch.cuda.is_available())"

[flckv]

I suspect that deepspeed is not working because I am using a SLURM on premise cluster for it? @loadams is there a guide how to install deepspeed in conda on cluster login node then be able to from bash script call the deepspeed on other gpu nodes ?

[fecet]

#3067 (comment) can you confirm if this conda environment works?

It worked for me before, but I cannot use it with latest deepspeed

[loadams]

Hi @flckv - there is a slurm supported launcher and runner in DeepSpeed, so that shouldn't be the problem. You should be able to install DeepSpeed as you normally would on the login node, then specify the gpu resources to run on in the hostfile and use your other gpu nodes..

[loadams]

@fecet - that looks to be a different issue to me, if you are still having issues can you open a new issue and link your original one, please?