Support CUDA 12.9

[cailun01]

Feature request

Hi, bitsandbytes team!

I installed bitsandbytes==4.46.0 in a CUDA 12.9 environment. However, when I tried to import bitsandbytes, I encountered the following error:

bitsandbytes library load error: Configured CUDA binary not found at /usr/local/lib/python3.12/dist-packages/bitsandbytes/libbitsandbytes_cuda129.so
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/bitsandbytes/cextension.py", line 290, in <module>
    lib = get_native_library()
          ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/bitsandbytes/cextension.py", line 270, in get_native_library
    raise RuntimeError(f"Configured CUDA binary not found at {cuda_binary_path}")
RuntimeError: Configured CUDA binary not found at /usr/local/lib/python3.12/dist-packages/bitsandbytes/libbitsandbytes_cuda129.so

Looking at the installation directory (/usr/local/lib/python3.12/dist-packages/bitsandbytes/), I noticed that there is no CUDA 12.9 binary present:

__init__.py  _ops.py   cextension.py  diagnostics             libbitsandbytes_cuda118.so  libbitsandbytes_cuda122.so  libbitsandbytes_cuda125.so  nn        triton
__main__.py  autograd  consts.py      functional.py           libbitsandbytes_cuda120.so  libbitsandbytes_cuda123.so  libbitsandbytes_cuda126.so  optim     utils.py
__pycache__  backends  cuda_specs.py  libbitsandbytes_cpu.so  libbitsandbytes_cuda121.so  libbitsandbytes_cuda124.so  libbitsandbytes_cuda128.so  research

Do you have any plans to support CUDA 12.9 in an upcoming release? Can I build bitsandbytes from source by master branch to support CUDA 12.9?

Thanks!

Motivation

Support to newest CUDA version to support sm100 and sm120.

Your contribution

No

[cailun01]

I tried to compile bnb(commit: aca9778e742980e904ee10b417e219925e6c249d) from source but got this error:

In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/util_arch.cuh:47,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/config.cuh:46,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/device/device_scan.cuh:35,
                 from /root/bitsandbytes/csrc/ops.cu:9:
/usr/local/cuda/targets/sbsa-linux/include/cub/util_macro.cuh:50:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   50 | CUB_NAMESPACE_BEGIN
      | ^~~~~
In file included from /usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/assert.h:28,
                 from /usr/local/cuda/targets/sbsa-linux/include/cuda/__cccl_config:14,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/config.cuh:36:
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/util_arch.cuh:53:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   53 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/device/device_scan.cuh:45:
/usr/local/cuda/targets/sbsa-linux/include/cub/detail/choose_offset.cuh:46:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   46 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/device/device_scan.cuh:46:
/usr/local/cuda/targets/sbsa-linux/include/cub/detail/nvtx.cuh:67:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   67 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/util_ptx.cuh:46,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/detail/strong_load.cuh:45,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/agent/single_pass_scan_operators.cuh:46,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/agent/agent_scan.cuh:46,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/device/dispatch/dispatch_scan.cuh:48,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/device/device_scan.cuh:47:
/usr/local/cuda/targets/sbsa-linux/include/cub/util_debug.cuh:174:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
  174 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/util_type.cuh:48,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/util_ptx.cuh:47:
/usr/local/cuda/targets/sbsa-linux/include/cub/detail/type_traits.cuh:59:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   59 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/util_type.cuh:49:
/usr/local/cuda/targets/sbsa-linux/include/cub/detail/uninitialized_copy.cuh:43:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   43 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/iterator_adaptor.h:43,
                 from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/counting_iterator.h:43,
                 from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/discard_iterator_base.h:28,
                 from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/discard_iterator.h:32,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/util_type.cuh:51:
/usr/local/cuda/targets/sbsa-linux/include/thrust/detail/use_default.h:29:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   29 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/iterator_adaptor_base.h:29,
                 from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/iterator_adaptor.h:44:
/usr/local/cuda/targets/sbsa-linux/include/thrust/detail/type_traits.h:36:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   36 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/iterator_facade.h:44,
                 from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/iterator_adaptor_base.h:31:
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/distance_from_result.h:30:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   30 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/any_system_tag.h:28,
                 from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/iterator_facade_category.h:29,
                 from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/iterator_facade.h:45:
/usr/local/cuda/targets/sbsa-linux/include/thrust/detail/execution_policy.h:29:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   29 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/any_system_tag.h:30:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   30 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/thrust/system/cuda/detail/execution_policy.h:40,
                 from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/device_system_tag.h:31,
                 from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/iterator_facade_category.h:30:
/usr/local/cuda/targets/sbsa-linux/include/thrust/system/cuda/config.h:126:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
  126 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/system/cuda/config.h:149:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
  149 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/thrust/detail/allocator_aware_execution_policy.h:28,
                 from /usr/local/cuda/targets/sbsa-linux/include/thrust/system/cuda/detail/execution_policy.h:42:
/usr/local/cuda/targets/sbsa-linux/include/thrust/detail/alignment.h:36:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   36 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/thrust/detail/execute_with_allocator_fwd.h:31,
                 from /usr/local/cuda/targets/sbsa-linux/include/thrust/detail/allocator_aware_execution_policy.h:29:
/usr/local/cuda/targets/sbsa-linux/include/thrust/detail/execute_with_dependencies.h:36:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   36 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/detail/execute_with_allocator_fwd.h:34:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   34 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/detail/allocator_aware_execution_policy.h:33:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   33 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/thrust/system/cuda/detail/execution_policy.h:48:
/usr/local/cuda/targets/sbsa-linux/include/thrust/detail/dependencies_aware_execution_policy.h:33:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   33 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/system/cuda/detail/execution_policy.h:51:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   51 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/device_system_tag.h:34:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   34 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/thrust/system/cpp/detail/execution_policy.h:28,
                 from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/host_system_tag.h:31,
                 from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/iterator_facade_category.h:31:
/usr/local/cuda/targets/sbsa-linux/include/thrust/system/detail/sequential/execution_policy.h:30:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   30 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/system/cpp/detail/execution_policy.h:30:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   30 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/host_system_tag.h:34:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   34 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/iterator_categories.h:42,
                 from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/is_iterator_category.h:29,
                 from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/iterator_facade_category.h:32:
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/iterator_category_with_system_and_traversal.h:29:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   29 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/iterator_categories.h:43:
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/iterator_traversal_tags.h:29:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   29 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/iterator_categories.h:47:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   47 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/iterator_categories.h:217:
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/universal_categories.h:32:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   32 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/is_iterator_category.h:31:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   31 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/iterator_category_to_traversal.h:29,
                 from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/iterator_facade_category.h:33:
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/iterator_category_to_system.h:35:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   35 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/iterator_category_to_traversal.h:33:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   33 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/iterator_facade_category.h:40:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   40 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/iterator_facade.h:47:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   47 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/iterator_adaptor_base.h:32:
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/iterator_traits.h:48:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   48 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/iterator_traits.h:85:
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/iterator_traits.inl:35:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   35 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/iterator_adaptor_base.h:34:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   34 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/iterator_adaptor.h:47:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   47 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/counting_iterator.inl:29,
                 from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/counting_iterator.h:48:
/usr/local/cuda/targets/sbsa-linux/include/thrust/detail/numeric_traits.h:35:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   35 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/counting_iterator.inl:36:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   36 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/counting_iterator.h:50:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   50 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/discard_iterator_base.h:29:
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/any_assign.h:29:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   29 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/detail/discard_iterator_base.h:34:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   34 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/thrust/iterator/discard_iterator.h:38:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   38 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/util_type.cuh:81:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   81 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/util_ptx.cuh:49:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   49 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/detail/strong_load.cuh:50:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   50 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/agent/single_pass_scan_operators.cuh:47:
/usr/local/cuda/targets/sbsa-linux/include/cub/detail/strong_store.cuh:48:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   48 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/agent/single_pass_scan_operators.cuh:49:
/usr/local/cuda/targets/sbsa-linux/include/cub/thread/thread_load.cuh:51:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   51 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/agent/single_pass_scan_operators.cuh:50:
/usr/local/cuda/targets/sbsa-linux/include/cub/thread/thread_store.cuh:49:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   49 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/agent/single_pass_scan_operators.cuh:51:
/usr/local/cuda/targets/sbsa-linux/include/cub/util_temporary_storage.cuh:49:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   49 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/warp/warp_reduce.cuh:47,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/agent/single_pass_scan_operators.cuh:52:
/usr/local/cuda/targets/sbsa-linux/include/cub/thread/thread_operators.cuh:70:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   70 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/warp/warp_reduce.cuh:49:
/usr/local/cuda/targets/sbsa-linux/include/cub/warp/specializations/warp_reduce_shfl.cuh:57:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   57 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/warp/warp_reduce.cuh:50:
/usr/local/cuda/targets/sbsa-linux/include/cub/warp/specializations/warp_reduce_smem.cuh:54:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   54 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/warp/warp_reduce.cuh:54:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   54 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/agent/single_pass_scan_operators.cuh:60:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   60 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/warp/warp_exchange.cuh:48,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/block/block_exchange.cuh:48,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/block/block_load.cuh:44,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/agent/agent_scan.cuh:47:
/usr/local/cuda/targets/sbsa-linux/include/cub/warp/specializations/warp_exchange_shfl.cuh:45:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   45 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/warp/warp_exchange.cuh:49:
/usr/local/cuda/targets/sbsa-linux/include/cub/warp/specializations/warp_exchange_smem.cuh:51:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   51 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/warp/warp_exchange.cuh:53:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   53 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/block/block_exchange.cuh:52:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   52 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/block/block_load.cuh:45:
/usr/local/cuda/targets/sbsa-linux/include/cub/iterator/cache_modified_input_iterator.cuh:59:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   59 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/block/block_load.cuh:49:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   49 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/block/specializations/block_scan_raking.cuh:47,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/block/block_scan.cuh:45,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/agent/agent_scan.cuh:48:
/usr/local/cuda/targets/sbsa-linux/include/cub/block/block_raking_layout.cuh:49:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   49 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/thread/thread_reduce.cuh:44,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/block/specializations/block_scan_raking.cuh:49:
/usr/local/cuda/targets/sbsa-linux/include/cub/detail/array_utils.cuh:48:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   48 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/thread/thread_reduce.cuh:68:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   68 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/block/specializations/block_scan_raking.cuh:50:
/usr/local/cuda/targets/sbsa-linux/include/cub/thread/thread_scan.cuh:48:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   48 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/warp/warp_scan.cuh:49,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/block/specializations/block_scan_raking.cuh:52:
/usr/local/cuda/targets/sbsa-linux/include/cub/warp/specializations/warp_scan_shfl.cuh:53:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   53 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/warp/warp_scan.cuh:50:
/usr/local/cuda/targets/sbsa-linux/include/cub/warp/specializations/warp_scan_smem.cuh:54:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   54 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/warp/warp_scan.cuh:55:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   55 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/block/specializations/block_scan_raking.cuh:54:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   54 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/block/block_scan.cuh:46:
/usr/local/cuda/targets/sbsa-linux/include/cub/block/specializations/block_scan_warp_scans.cuh:52:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   52 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/block/block_scan.cuh:52:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   52 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/agent/agent_scan.cuh:49:
/usr/local/cuda/targets/sbsa-linux/include/cub/block/block_store.cuh:48:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   48 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/agent/agent_scan.cuh:50:
/usr/local/cuda/targets/sbsa-linux/include/cub/grid/grid_queue.cuh:50:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   50 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/agent/agent_scan.cuh:57:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   57 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/util_device.cuh:50,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/device/dispatch/tuning/tuning_scan.cuh:45,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/device/dispatch/dispatch_scan.cuh:49:
/usr/local/cuda/targets/sbsa-linux/include/cub/detail/device_synchronize.cuh:35:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   35 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/util_device.cuh:66:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   66 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/detail/launcher/cuda_runtime.cuh:15,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/util_device.cuh:807:
/usr/local/cuda/targets/sbsa-linux/include/thrust/system/cuda/detail/core/triple_chevron_launch.h:44:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   44 | THRUST_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/detail/launcher/cuda_runtime.cuh:17:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   17 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/device/dispatch/tuning/tuning_scan.cuh:51:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   51 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/device/dispatch/dispatch_scan.cuh:54:
/usr/local/cuda/targets/sbsa-linux/include/cub/util_math.cuh:50:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   50 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/device/dispatch/dispatch_scan.cuh:62:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   62 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/agent/agent_scan_by_key.cuh:46,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/device/dispatch/dispatch_scan_by_key.cuh:46,
                 from /usr/local/cuda/targets/sbsa-linux/include/cub/device/device_scan.cuh:48:
/usr/local/cuda/targets/sbsa-linux/include/cub/block/block_discontinuity.cuh:50:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   50 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/agent/agent_scan_by_key.cuh:57:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   57 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
In file included from /usr/local/cuda/targets/sbsa-linux/include/cub/device/dispatch/dispatch_scan_by_key.cuh:48:
/usr/local/cuda/targets/sbsa-linux/include/cub/device/dispatch/tuning/tuning_scan_by_key.cuh:50:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   50 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/device/dispatch/dispatch_scan_by_key.cuh:60:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   60 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
/usr/local/cuda/targets/sbsa-linux/include/cub/device/device_scan.cuh:53:1: error: macro "_CCCL_PP_SPLICE_WITH_IMPL1" passed 3 arguments, but takes just 2
   53 | CUB_NAMESPACE_BEGIN
      | ^~~~~
/usr/local/cuda/targets/sbsa-linux/include/cuda/std/__cccl/preprocessor.h:1200: note: macro "_CCCL_PP_SPLICE_WITH_IMPL1" defined here
 1200 | #define _CCCL_PP_SPLICE_WITH_IMPL1(SEP, P1)       P1
      | 
make[2]: *** [CMakeFiles/bitsandbytes.dir/build.make:122: CMakeFiles/bitsandbytes.dir/csrc/ops.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:87: CMakeFiles/bitsandbytes.dir/all] Error 2
make: *** [Makefile:91: all] Error 2

[matthewdouglas]

We do plan to release with binaries built against 12.9 in the near future. In fact, for development I've been testing with torch 2.8.0+cu129 nightlies. I haven't run into an issue building on x86-64 but will look into aarch64.

Is there a particular reason you need a CUDA 12.9 build at the moment? You should be able to get by in the meantime by using the CUDA 12.8 builds of PyTorch. If you must use a CUDA 12.9 PyTorch build, you can fallback to the CUDA 12.8 bitsandbytes binary with an environment variable: BNB_CUDA_VERSION=128 and still have support for sm100+.

[matthewdouglas]

The build issue appears related to this: NVIDIA/cccl#4967

I would recommend limiting the build targets if you do build from source, i.e. for Blackwell:

cmake -DCOMPUTE_BACKEND=cuda -DCOMPUTE_CAPABILITY="100;120"

[cailun01]

The build issue appears related to this: NVIDIA/cccl#4967

I would recommend limiting the build targets if you do build from source, i.e. for Blackwell:

cmake -DCOMPUTE_BACKEND=cuda -DCOMPUTE_CAPABILITY="100;120"

@matthewdouglas Hi, matthewdouglas. Thanks for your support. I have to use CUDA 12.9 due to customer requirements.

I built from source with cmake -DCOMPUTE_BACKEND=cuda -DCOMPUTE_CAPABILITY="100;120". But got this error when import bitsandbytes:

bitsandbytes library load error: /usr/local/lib/python3.12/dist-packages/bitsandbytes/libbitsandbytes_cpu.so: cannot open shared object file: No such file or directory
 If you are using Intel CPU/XPU, please install intel_extension_for_pytorch to enable required ops
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/bitsandbytes/cextension.py", line 319, in <module>
    lib = get_native_library()
          ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/bitsandbytes/cextension.py", line 289, in get_native_library
    dll = ct.cdll.LoadLibrary(str(binary_path))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/ctypes/__init__.py", line 460, in LoadLibrary
    return self._dlltype(name)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/ctypes/__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /usr/local/lib/python3.12/dist-packages/bitsandbytes/libbitsandbytes_cpu.so: cannot open shared object file: No such file or directory

Looking at the installation directory (/usr/local/lib/python3.12/dist-packages/bitsandbytes/), I noticed that there is no libbitsandbytes_cpu.so present:

__init__.py  __main__.py  __pycache__  _ops.py  autograd  backends  cextension.py  consts.py  cuda_specs.py  diagnostics  functional.py  libbitsandbytes_cuda129.so  nn  optim  research  triton  utils.py

Would you mind tell me how to build libbitsandbytes_cpu.so?

[cailun01]

What confuses me is that I have compiled the binary files for both CPU and CUDA separately, but after importing bitsandbytes, I encountered this log:

The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.

I installed bitsandbytes by:

mkdir build_cuda
cmake -S . -B build_cuda -DCOMPUTE_BACKEND=cuda -DCOMPUTE_CAPABILITY="100;120"
cmake --build build_cuda --config Release

mkdir build_cpu
cmake -S . -B build_cpu -DCOMPUTE_BACKEND=cpu
cmake --build build_cpu --config Release
python3 setup.py bdist_wheel
pip3 install dist/*.whl

[cailun01]

What confuses me is that I have compiled the binary files for both CPU and CUDA separately, but after importing bitsandbytes, I encountered this log:

The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers and GPU quantization are unavailable.

I installed bitsandbytes by:

mkdir build_cuda
cmake -S . -B build_cuda -DCOMPUTE_BACKEND=cuda -DCOMPUTE_CAPABILITY="100;120"
cmake --build build_cuda --config Release

mkdir build_cpu
cmake -S . -B build_cpu -DCOMPUTE_BACKEND=cpu
cmake --build build_cpu --config Release
python3 setup.py bdist_wheel
pip3 install dist/*.whl

I mistakenly installed an incorrect version of PyTorch, which led to the problem. The issue has now been resolved.