Xccl process group for Pytorch by Chao1Han · Pull Request #1 · Chao1Han/pytorch

Chao1Han · 2024-09-03T01:42:05Z

No description provided.

zhangxiaoli73 · 2024-09-03T01:46:08Z

You missed xccl register in https://github.com/Chao1Han/pytorch/blob/master/torch/distributed/distributed_c10d.py

zhangxiaoli73 · 2024-09-09T02:24:18Z

There is some record function call in ProcessGroupNCCL. Could you check whether those needed on XPU? We need to make sure basic profiler can work on XPU distributed.

Chao1Han · 2024-09-09T10:00:42Z

@gujinghui @guangyey @zhangxiaoli73 pls help review again.

zhangxiaoli73 · 2024-09-10T02:52:53Z

+    TORCH_CHECK(false, "ProcessGroupXCCL::broadcast not implemented");
+  }
+
+  c10::intrusive_ptr<Work> allreduce_sparse(


We don't have plan to support sparse. Then could we skip this op in your ops register for XPU?

removed

We don't have plan to support sparse. Then could we skip this op in your ops register for XPU?

gujinghui · 2024-09-11T06:29:37Z

LGTM. Next step is to add UTs?

Chao1Han · 2024-09-11T06:34:33Z

LGTM. Next step is to add UTs?

Yes, will create simple ut in test/distributed/test_c10d_xccl.py since just enable allreduce.

zhangxiaoli73 · 2024-09-11T06:37:23Z

+    target_compile_definitions(torch_xpu PUBLIC USE_C10D_XCCL)
+    set_source_files_properties(
+      ${TORCH_SRC_DIR}/csrc/distributed/c10d/ProcessGroupXCCL.cpp
+      PROPERTIES COMPILE_DEFINITIONS  "CCL_ENABLE_ZE;CCL_ENABLE_SYCL")


Those flags are just workaround and will be removed after oneCCL gcc build issue is fixed (target to basekit 2025.0).

zhangxiaoli73 · 2024-09-11T07:16:10Z

@Chao1Han Please make sure your PR include: 1) non-MPI launcher, please pass correct env options to oneCCL 2) Error handling. Abort the work(may need to call some oneCCL finalize API) for timeout.

zhangxiaoli73 · 2024-09-11T07:21:33Z

LGTM. Next step is to add UTs?

Add UT and also some implementation details refine (see my comments). @Chao1Han

zhangxiaoli73 · 2024-09-13T06:22:22Z

+  blockingWait_ = getCvarBool(TORCH_XCCL_BLOCKING_WAIT, false);
+  init();
+
+  {


You need to check its launcher is IMPI or not. Only set CCL_LOCAL_RANK/CCL_LOCAL_SIZE for non-MPI launcher

zhangxiaoli73 · 2024-09-19T08:30:50Z

+  std::lock_guard<std::mutex> lock(kvs_mutex);
+  if (kvs)
+    return kvs;
+  std::string storeKey = "ccl_kvs";


…rch#136266) This fixes a subset of issues for dynamic shapes + DTensor. It's pretty easy to run into other issues - it's likely that we need pytorch#125941 to land for DTensor + dynamic shapes to work more generally. I ended up writing a test that had dynamic shape inputs but not dynamic shape outputs in order to properly test this fix Pull Request resolved: pytorch#136266 Approved by: https://github.com/ezyang, https://github.com/yf225

…a 12.6 (pytorch#136321) To prepare for future cuda updates. Pull Request resolved: pytorch#136321 Approved by: https://github.com/Skylion007, https://github.com/eqy

Summary: This logs all operations when tracing log level is enabled for the `TCPStoreLibUvBackend`. This is very useful for debugging collective operations when issues occur as it logs all hosts and the keys that they're modifying. To minimize total data we only log the keys and not the values This changes the C10D_* macros to be much more efficient -- previously we would always format the log string even if they would never be printed which is very wasteful for detailed tracing. This now gates them with an if statement to achieve the same behavior with no overhead Test Plan: ``` TORCH_DISTRIBUTED_DEBUG=DETAIL torchrun --nnodes 1 --nproc_per_node 1 --no-python /bin/bash -c "echo foo" ``` ``` I0919 09:26:52.352013 34271 TCPStore.cpp:285] [c10d - debug] The server has started on port = 29500. I0919 09:26:52.352246 34271 socket.cpp:783] [c10d - debug] The client socket will attempt to connect to an IPv6 address of (127.0.0.1, 29500). I0919 09:26:52.352241 36903 TCPStoreLibUvBackend.cpp:1173] [c10d - debug] Uv main loop running I0919 09:26:52.352308 34271 socket.cpp:854] [c10d - trace] The client socket is attempting to connect to [localhost]:29500. I0919 09:26:52.353633 34271 socket.cpp:945] [c10d] The client socket has connected to [localhost]:29500 on SocketImpl(fd=41, addr=[localhost]:45646, remote=[localhost]:29500). I0919 09:26:52.354422 34271 TCPStore.cpp:321] [c10d - debug] TCP client connected to host 127.0.0.1:29500 I0919 09:26:52.354558 36903 TCPStoreLibUvBackend.cpp:774] [c10d - trace] validate magic:1015412686 address:[localhost]:45646 I0919 09:26:52.354638 36903 TCPStoreLibUvBackend.cpp:789] [c10d - trace] ping nonce:34271 address:[localhost]:45646 I0919 09:26:52.356122 36903 TCPStoreLibUvBackend.cpp:866] [c10d - trace] add key:init/ val:1 address:[localhost]:45646 I0919 09:26:52.356308 36903 TCPStoreLibUvBackend.cpp:930] [c10d - trace] wait key_count:1 address:[localhost]:45646 I0919 09:26:52.356410 36903 TCPStoreLibUvBackend.cpp:846] [c10d - trace] get key:init/ address:[localhost]:45646 I0919 09:26:52.358688 36903 TCPStoreLibUvBackend.cpp:808] [c10d - trace] set key:/none/torchelastic/role_info/0 address:[localhost]:45646 I0919 09:26:52.360177 36903 TCPStoreLibUvBackend.cpp:930] [c10d - trace] wait key_count:1 address:[localhost]:45646 I0919 09:26:52.360296 36903 TCPStoreLibUvBackend.cpp:1004] [c10d - trace] multi_get key_count:1 address:[localhost]:45646 I0919 09:26:52.362076 36903 TCPStoreLibUvBackend.cpp:1036] [c10d - trace] multi_set key_count:1 address:[localhost]:45646 I0919 09:26:52.364001 36903 TCPStoreLibUvBackend.cpp:930] [c10d - trace] wait key_count:1 address:[localhost]:45646 I0919 09:26:52.364091 36903 TCPStoreLibUvBackend.cpp:846] [c10d - trace] get key:/none/torchelastic/assigned_ranks/0 address:[localhost]:45646 ``` Differential Revision: D62924454 Pull Request resolved: pytorch#136320 Approved by: https://github.com/c-p-i-o, https://github.com/XilunWu

…ers for a better debugging experience (pytorch#135913) Summary: The change involves passing the expired timers to the log_debug_info_for_expired_timers function after to_json() has been applied . This change is made to provide a better debugging experience for the user. Test Plan: unit tests Reviewed By: gag1jain Differential Revision: D62408767 Pull Request resolved: pytorch#135913 Approved by: https://github.com/gag1jain

@zdevito

…ytorch#139659) ### Motivation Today, watchdog only reports that it found a collective timeout: ``` [rank1]:[E1104 14:02:18.767594328 ProcessGroupNCCL.cpp:688] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=1, OpType=ALLREDUCE, NumelIn=200, NumelOut=200, Timeout(ms)=5000) ran for 5096 milliseconds before timing out. ``` While this is nice, it is hard to associate the error with user's program or library stack. ### This PR This PR gives watchdog the ability to report the call-time stack of the collective, so that it would be easier to track the error back to the program's behavior. The call-time stack was recorded by Flight Recorder with minimal overhead (for details, please read this [doc](https://dev-discuss.pytorch.org/t/fast-combined-c-python-torchscript-inductor-tracebacks/1158) written by @zdevito ). In `ProcessGroupNCCL`, we are only tracking / reporting the python part so that it fits most PyTorch users. ### Demo [stack_demo.py](https://gist.github.com/kwen2501/6758e18d305d67fc6f3f926217825c09). ``` TORCH_NCCL_TRACE_BUFFER_SIZE=100 torchrun --nproc-per-node 2 stack_demo.py ``` `TORCH_NCCL_TRACE_BUFFER_SIZE` is for turning on the Flight Recorder. Output: ``` [rank0]:[E1104 14:19:27.591610653 ProcessGroupNCCL.cpp:695] Stack trace of the timedout collective operation: #0 all_reduce from /data/users/kw2501/pytorch/torch/distributed/distributed_c10d.py:2696 #1 wrapper from /data/users/kw2501/pytorch/torch/distributed/c10d_logger.py:83 #2 bar from /data/users/kw2501/sync_async/repro.py:15 #3 foo from /data/users/kw2501/sync_async/repro.py:24 #4 main from /data/users/kw2501/sync_async/repro.py:34 #5 <module> from /data/users/kw2501/sync_async/repro.py:40 [rank1]:[E1104 14:19:27.771430164 ProcessGroupNCCL.cpp:695] Stack trace of the timedout collective operation: #0 all_gather_into_tensor from /data/users/kw2501/pytorch/torch/distributed/distributed_c10d.py:3630 #1 wrapper from /data/users/kw2501/pytorch/torch/distributed/c10d_logger.py:83 #2 baz from /data/users/kw2501/sync_async/repro.py:20 #3 foo from /data/users/kw2501/sync_async/repro.py:26 #4 main from /data/users/kw2501/sync_async/repro.py:34 #5 <module> from /data/users/kw2501/sync_async/repro.py:40 ``` From the log above, we can tell that `bar()` and `baz()` are the places where the two ranks divert. Pull Request resolved: pytorch#139659 Approved by: https://github.com/wconstab, https://github.com/fduwjj

See pytorch#140725 (comment) Running `torch.mps.synchronize()` after metal kernel resulted in infinite wait inside `[_MTLCommandBuffer waitUntilCompleted]` ``` (lldb) bt * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP * frame #0: 0x00000001aa919084 Metal`pthread_cond_wait + 12 frame #1: 0x00000001aa78b1b4 Metal`-[_MTLCommandBuffer waitUntilCompleted] + 84 frame #2: 0x00000001032bf358 libtorch_python.dylib`torch::mps::MPSModule_deviceSynchronize(_object*, _object*) + 40 frame #3: 0x0000000100e94c20 Python`cfunction_vectorcall_NOARGS + 100 frame #4: 0x0000000100e389b8 Python`PyObject_Vectorcall + 92 frame #5: 0x0000000100f61e38 Python`_PyEval_EvalFrameDefault + 19040 frame #6: 0x0000000100f5d180 Python`PyEval_EvalCode + 200 frame #7: 0x0000000100fcd1a4 Python`run_eval_code_obj + 104 frame #8: 0x0000000100fccbe4 Python`run_mod + 168 frame #9: 0x0000000100fcb518 Python`pyrun_file + 164 frame #10: 0x0000000100fca854 Python`_PyRun_SimpleFileObject + 256 frame #11: 0x0000000100fca4e8 Python`_PyRun_AnyFileObject + 80 frame #12: 0x0000000100ff2028 Python`pymain_run_file_obj + 164 frame #13: 0x0000000100ff1ce4 Python`pymain_run_file + 72 frame #14: 0x0000000100ff0f74 Python`Py_RunMain + 988 frame #15: 0x0000000100ff1564 Python`pymain_main + 304 frame #16: 0x0000000100ff1604 Python`Py_BytesMain + 40 frame #17: 0x000000019f630274 dyld`start + 2840 ``` Pull Request resolved: pytorch#141296 Approved by: https://github.com/huydhn

…143550) # Motivation Fix pytorch#143543 # Solution We should raise python exception instead of aborting... # Additional Context without this PR: ```python >>> import torch >>> torch.accelerator.current_stream(torch.accelerator.device_count()) terminate called after throwing an instance of 'c10::Error' what(): device is out of range, device is 2, total number of device is 2. Exception raised from check_device_index at /home/dvrogozh/git/pytorch/pytorch/c10/xpu/XPUFunctions.h:36 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xac (0x7f30707eb95c in /home/dvrogozh/git/pytorch/pytorch/torch/lib/libc10.so) frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xf3 (0x7f307078fc57 in /home/dvrogozh/git/pytorch/pytorch/torch/lib/libc10.so) frame #2: <unknown function> + 0x19a3e (0x7f3070c2ba3e in /home/dvrogozh/git/pytorch/pytorch/torch/lib/libc10_xpu.so) frame #3: c10::xpu::getCurrentXPUStream(signed char) + 0x2f (0x7f3070c2c83f in /home/dvrogozh/git/pytorch/pytorch/torch/lib/libc10_xpu.so) frame #4: <unknown function> + 0x1ca35 (0x7f3070c2ea35 in /home/dvrogozh/git/pytorch/pytorch/torch/lib/libc10_xpu.so) frame #5: <unknown function> + 0x653f15 (0x7f3083391f15 in /home/dvrogozh/git/pytorch/pytorch/torch/lib/libtorch_python.so) frame #6: <unknown function> + 0x39e5f2 (0x7f30830dc5f2 in /home/dvrogozh/git/pytorch/pytorch/torch/lib/libtorch_python.so) <omitting python frames> frame #20: <unknown function> + 0x29d90 (0x7f308b19bd90 in /lib/x86_64-linux-gnu/libc.so.6) frame #21: __libc_start_main + 0x80 (0x7f308b19be40 in /lib/x86_64-linux-gnu/libc.so.6) Aborted (core dumped) ``` with this PR: ```python >>> import torch >>> torch.accelerator.current_stream(torch.accelerator.device_count()) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/pt-gpu/4T-4652/guangyey/stock-pytorch/torch/accelerator/__init__.py", line 123, in current_stream return torch._C._accelerator_getStream(device_index) RuntimeError: The device index is out of range. It must be in [0, 2), but got 2. ``` Pull Request resolved: pytorch#143550 Approved by: https://github.com/EikanWang, https://github.com/dvrogozh, https://github.com/albanD

…pytorch#144120) (pytorch#146372) Summary: # Summary ### Sticky points Cuda-graph rng handling has changed / deviated from original implementation. We will be left with a dangling 'offset' val and confusing naming due to BC ## Dependencies - Flash PR: Dao-AILab/flash-attention#1419 ### Other Points - The BC linter is complaining about losing generate.py and its functions which is not real BC surface cc albanD imported-using-ghimport Test Plan: Imported from OSS Building in dev `buck build @//mode/dev-nosan -c fbcode.nvcc_arch=h100a //caffe2:ATen-cu --show-full-output ` I and Nming the .so I do see that the flash symbols are correctly named: ``` 0000000001c3dfb0 t pytorch_flash::run_mha_bwd(pytorch_flash::Flash_bwd_params&, CUstream_st*)::$_0::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const::{lambda()#7}::operator()() const 0000000001c36080 t pytorch_flash::run_mha_fwd(pytorch_flash::Flash_fwd_params&, CUstream_st*, bool)::$_0::operator()() const::{lambda()#2}::operator()() const::{lambda()#1}::operator()() const::{lambda()#6}::operator()() const 0000000001c360e0 t pytorch_flash::run_mha_fwd(pytorch_flash::Flash_fwd_params&, CUstream_st*, bool)::$_0::operator()() const::{lambda()#2}::operator()() const::{lambda()#1}::operator()() const::{lambda()#7}::operator()() const 0000000001c35fc0 t pytorch_flash::run_mha_fwd(pytorch_flash::Flash_fwd_params&, CUstream_st*, bool)::$_0::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const::{lambda()#6}::operator()() const 0000000001c36020 t pytorch_flash::run_mha_fwd(pytorch_flash::Flash_fwd_params&, CUstream_st*, bool)::$_0::operator()() const::{lambda()#1}::operator()() const::{lambda()#1}::operator()() const::{lambda()#7}::operator()() const ``` Reviewed By: vkuzo Differential Revision: D68502879 Pulled By: drisspg Pull Request resolved: pytorch#146372 Approved by: https://github.com/jbschlosser

Which inherits from `RuntimeError` and contains `error_code`, which in case of CUDA should contain error returned by `cudaGetLastError` `torch::detail::_new_accelerator_error_object(c10::AcceleratorError&)` follows the pattern of CPython's [`PyErr_SetString`](https://github.com/python/cpython/blob/cb8a72b301f47e76d93a7fe5b259e9a5758792e1/Python/errors.c#L282), namely - Convert cstr into Python string with `PyUnicode_FromString` - Create new exception object using `PyObject_CallOneArg` just like it's done in [`_PyErr_CreateException`](https://github.com/python/cpython/blob/cb8a72b301f47e76d93a7fe5b259e9a5758792e1/Python/errors.c#L32) - Set `error_code` property using `PyObject_SetAttrString` - decref all temporary references Test that it works and captures CPP backtrace (in addition to CI) by running ```python import os os.environ['TORCH_SHOW_CPP_STACKTRACES'] = '1' import torch x = torch.rand(10, device="cuda") y = torch.arange(20, device="cuda") try: x[y] = 2 print(x) except torch.AcceleratorError as e: print("Exception was raised", e.args[0]) print("Captured error code is ", e.error_code) ``` which produces following output ``` Exception was raised CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. Exception raised from c10_cuda_check_implementation at /home/ubuntu/pytorch/c10/cuda/CUDAException.cpp:41 (most recent call first): C++ CapturedTraceback: #4 std::_Function_handler<std::shared_ptr<c10::LazyValue<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const> (), c10::SetStackTraceFetcher(std::function<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) from Logging.cpp:0 #5 c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) from ??:0 #6 c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) [clone .cold] from CUDAException.cpp:0 #7 void at::native::gpu_kernel_impl<at::native::AbsFunctor<float> >(at::TensorIteratorBase&, at::native::AbsFunctor<float> const&) [clone .isra.0] from tmpxft_000191fc_00000000-6_AbsKernel.cudafe1.cpp:0 #8 at::native::abs_kernel_cuda(at::TensorIteratorBase&) from ??:0 #9 at::Tensor& at::native::unary_op_impl_with_complex_to_float_out<at::native::abs_stub_DECLARE_DISPATCH_type>(at::Tensor&, at::Tensor const&, at::native::abs_stub_DECLARE_DISPATCH_type&, bool) [clone .constprop.0] from UnaryOps.cpp:0 #10 at::(anonymous namespace)::(anonymous namespace)::wrapper_CUDA_out_abs_out(at::Tensor const&, at::Tensor&) from RegisterCUDA_0.cpp:0 #11 at::_ops::abs_out::call(at::Tensor const&, at::Tensor&) from ??:0 #12 at::native::abs(at::Tensor const&) from ??:0 #13 c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&), &at::(anonymous namespace)::(anonymous namespace)::wrapper_CompositeExplicitAutograd__abs>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&> >, at::Tensor (at::Tensor const&)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&) from RegisterCompositeExplicitAutograd_0.cpp:0 #14 at::_ops::abs::redispatch(c10::DispatchKeySet, at::Tensor const&) from ??:0 #15 torch::autograd::VariableType::(anonymous namespace)::abs(c10::DispatchKeySet, at::Tensor const&) from VariableType_1.cpp:0 #16 c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (c10::DispatchKeySet, at::Tensor const&), &torch::autograd::VariableType::(anonymous namespace)::abs>, at::Tensor, c10::guts::typelist::typelist<c10::DispatchKeySet, at::Tensor const&> >, at::Tensor (c10::DispatchKeySet, at::Tensor const&)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&) from VariableType_1.cpp:0 #17 at::_ops::abs::call(at::Tensor const&) from ??:0 #18 at::native::isfinite(at::Tensor const&) from ??:0 #19 c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&), &at::(anonymous namespace)::(anonymous namespace)::wrapper_CompositeImplicitAutograd__isfinite>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&> >, at::Tensor (at::Tensor const&)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&) from RegisterCompositeImplicitAutograd_0.cpp:0 #20 at::_ops::isfinite::call(at::Tensor const&) from ??:0 #21 torch::autograd::THPVariable_isfinite(_object*, _object*, _object*) from python_torch_functions_2.cpp:0 #22 PyObject_CallFunctionObjArgs from ??:0 #23 _PyObject_MakeTpCall from ??:0 #24 _PyEval_EvalFrameDefault from ??:0 #25 _PyObject_FastCallDictTstate from ??:0 #26 _PyStack_AsDict from ??:0 pytorch#27 _PyObject_MakeTpCall from ??:0 pytorch#28 _PyEval_EvalFrameDefault from ??:0 pytorch#29 _PyFunction_Vectorcall from ??:0 pytorch#30 _PyEval_EvalFrameDefault from ??:0 pytorch#31 _PyFunction_Vectorcall from ??:0 pytorch#32 _PyEval_EvalFrameDefault from ??:0 pytorch#33 _PyFunction_Vectorcall from ??:0 pytorch#34 _PyEval_EvalFrameDefault from ??:0 pytorch#35 PyFrame_GetCode from ??:0 pytorch#36 PyNumber_Xor from ??:0 pytorch#37 PyObject_Str from ??:0 pytorch#38 PyFile_WriteObject from ??:0 pytorch#39 _PyWideStringList_AsList from ??:0 pytorch#40 _PyDict_NewPresized from ??:0 pytorch#41 _PyEval_EvalFrameDefault from ??:0 pytorch#42 PyEval_EvalCode from ??:0 pytorch#43 PyEval_EvalCode from ??:0 pytorch#44 PyUnicode_Tailmatch from ??:0 pytorch#45 PyInit__collections from ??:0 pytorch#46 PyUnicode_Tailmatch from ??:0 pytorch#47 _PyRun_SimpleFileObject from ??:0 pytorch#48 _PyRun_AnyFileObject from ??:0 pytorch#49 Py_RunMain from ??:0 pytorch#50 Py_BytesMain from ??:0 pytorch#51 __libc_init_first from ??:0 pytorch#52 __libc_start_main from ??:0 pytorch#53 _start from ??:0 Captured error code is 710 ``` Pull Request resolved: pytorch#152023 Approved by: https://github.com/eqy, https://github.com/mradmila, https://github.com/ngimel ghstack dependencies: pytorch#154436

Use uint64_t index types to avoid ``` torch_np/numpy_tests/core/test_einsum.py::TestEinsum::test_einsum_broadcast /var/lib/jenkins/workspace/aten/src/ATen/native/cpu/BlasKernel.cpp:132:24: runtime error: signed integer overflow: 9223365439786057728 + 13194139533312 cannot be represented in type 'long' #0 0x7f30d26166ba in std::enable_if<std::is_same_v<long, long>, void>::type at::native::cpublas::(anonymous namespace)::gemm_notrans_<long, long, long>(long, long, long, long, long const*, long, long const*, long, long, long*, long) /var/lib/jenkins/workspace/aten/src/ATen/native/cpu/BlasKernel.cpp:132:24 #1 0x7f30d26166ba in void at::native::cpublas::(anonymous namespace)::gemm_core_<long, long, long>(at::native::TransposeType, at::native::TransposeType, long, long, long, long, long const*, long, long const*, long, long, long*, long) /var/lib/jenkins/workspace/aten/src/ATen/native/cpu/BlasKernel.cpp:451:12 #2 0x7f30d25fba1b in at::native::cpublas::(anonymous namespace)::cpublas_gemm_impl(c10::ScalarType, at::native::TransposeType, at::native::TransposeType, long, long, long, c10::Scalar const&, void const*, long, void const*, long, c10::Scalar const&, void*, long)::$_2::operator()() const::'lambda2'()::operator()() const /var/lib/jenkins/workspace/aten/src/ATen/native/cpu/BlasKernel.cpp:485:3 #3 0x7f30d25fba1b in at::native::cpublas::(anonymous namespace)::cpublas_gemm_impl(c10::ScalarType, at::native::TransposeType, at::native::TransposeType, long, long, long, c10::Scalar const&, void const*, long, void const*, long, c10::Scalar const&, void*, long)::$_2::operator()() const /var/lib/jenkins/workspace/aten/src/ATen/native/cpu/BlasKernel.cpp:485:3 ``` Pull Request resolved: pytorch#154809 Approved by: https://github.com/soulitzer

Vibe-coded with Codex, after collecting a backtrace, see https://chatgpt.com/s/cd_68438be8a1248191adbfa0a5f000e60b Even though, check for empty tensor list exists in `at::cat` crash might happens while resolving named dimension to position, by calling `dimname_to_position(tensors[0], dim)`, see backtrace below ``` (lldb) up frame #1: 0x00000001101146dc libtorch_cpu.dylib`at::TensorBase::has_names(this=0x0000000000000000) const at TensorBase.h:559:10 556 bool has_names() const { 557 // If a user is using unnamed tensors, then we can short-circuit right here. 558 // Otherwise, impl::has_names attempts to retrieve names. -> 559 if (!impl_->has_named_tensor_meta()) { 560 return false; 561 } 562 return impl::has_names(unsafeGetTensorImpl()); (lldb) up frame #2: 0x00000001101144c4 libtorch_cpu.dylib`at::dimname_to_position(tensor=0x0000000000000000, dim=Dimname @ 0x000000016fdfe348) at NamedTensorUtils.cpp:23:3 20 int64_t dimname_to_position(const Tensor& tensor, Dimname dim) { 21 TORCH_CHECK(dim.type() != NameType::WILDCARD, 22 "Please look up dimensions by name, got: name = None."); -> 23 TORCH_CHECK(tensor.has_names(), 24 "Name ", dim, " not found in ", toDimnameRepr(tensor), "."); 25 const auto names = tensor.names(); 26 ``` TODOs: - May be move test from `test_tensor_creation.py` to OpInfo (not sure which one is more readable) - Replace `TORCH_CHECK` with `TORCH_CHECK_VALUE` and adjust unit tests Fixes pytorch#155306 Pull Request resolved: pytorch#155383 Approved by: https://github.com/cyyever, https://github.com/ezyang ghstack dependencies: pytorch#155382

…torch#156600) Don't call `sum()` on a tensor that is default constructed. Previously we could call `sum()` on a tensor that was default-contructed. That would lead to an error like this: ``` Traceback (most recent call last): File "/home/ahmads/.conda/envs/pt3/lib/python3.12/unittest/case.py", line 58, in testPartExecutor yield File "/home/ahmads/.conda/envs/pt3/lib/python3.12/unittest/case.py", line 634, in run self._callTestMethod(testMethod) File "/home/ahmads/.conda/envs/pt3/lib/python3.12/unittest/case.py", line 589, in _callTestMethod if method() is not None: ^^^^^^^^ File "/home/ahmads/personal/pytorch/torch/testing/_internal/common_utils.py", line 3191, in wrapper method(*args, **kwargs) File "/home/ahmads/personal/pytorch/test/test_nn.py", line 7235, in test_layer_norm_backwards_eps ln_out_cuda.backward(grad_output_cuda) File "/home/ahmads/personal/pytorch/torch/_tensor.py", line 647, in backward torch.autograd.backward( File "/home/ahmads/personal/pytorch/torch/autograd/__init__.py", line 354, in backward _engine_run_backward( File "/home/ahmads/personal/pytorch/torch/autograd/graph.py", line 829, in _engine_run_backward return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: tensor does not have a device Exception raised from device_default at /home/ahmads/personal/pytorch/c10/core/TensorImpl.h:1265 (most recent call first): C++ CapturedTraceback: #4 std::_Function_handler<std::shared_ptr<c10::LazyValue<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const> (), c10::SetStackTraceFetcher(std::function<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) from Logging.cpp:0 #5 c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) from ??:0 #6 c10::detail::torchCheckFail(char const*, char const*, unsigned int, char const*) from ??:0 #7 at::TensorBase::options() const from :0 #8 at::meta::resize_reduction(at::impl::MetaBase&, at::Tensor const&, c10::OptionalArrayRef<long>, bool, c10::ScalarType, bool) from :0 #9 at::meta::structured_sum_dim_IntList::meta(at::Tensor const&, c10::OptionalArrayRef<long>, bool, std::optional<c10::ScalarType>) from ??:0 #10 at::(anonymous namespace)::wrapper_CompositeExplicitAutogradNonFunctional_sum_dim_IntList(at::Tensor const&, c10::OptionalArrayRef<long>, bool, std::optional<c10::ScalarType>) from RegisterCompositeExplicitAutogradNonFunctional_0.cpp:0 #11 c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, c10::OptionalArrayRef<long>, bool, std::optional<c10::ScalarType>), &at::(anonymous namespace)::wrapper_CompositeExplicitAutogradNonFunctional_sum_dim_IntList>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, c10::OptionalArrayRef<long>, bool, std::optional<c10::ScalarType> > >, at::Tensor (at::Tensor const&, c10::OptionalArrayRef<long>, bool, std::optional<c10::ScalarType>)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, c10::OptionalArrayRef<long>, bool, std::optional<c10::ScalarType>) from RegisterCompositeExplicitAutogradNonFunctional_0.cpp:0 #12 at::_ops::sum_dim_IntList::call(at::Tensor const&, c10::OptionalArrayRef<long>, bool, std::optional<c10::ScalarType>) from ??:0 #13 void at::native::(anonymous namespace)::LaunchGammaBetaBackwardCUDAKernel<float, float>(float const*, float const*, float const*, float const*, long, long, at::Tensor*, at::Tensor*, CUstream_st*) from ??:0 #14 void at::native::(anonymous namespace)::LayerNormBackwardKernelImplInternal<float>(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, long, long, at::Tensor*, at::Tensor*, at::Tensor*) from ??:0 #15 at::native::(anonymous namespace)::LayerNormBackwardKernelImpl(at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, at::Tensor const&, long, long, at::Tensor*, at::Tensor*, at::Tensor*) from ??:0 #16 at::native::layer_norm_backward_cuda(at::Tensor const&, at::Tensor const&, c10::ArrayRef<long>, at::Tensor const&, at::Tensor const&, std::optional<at::Tensor> const&, std::optional<at::Tensor> const&, std::array<bool, 3ul>) from ??:0 #17 at::(anonymous namespace)::(anonymous namespace)::wrapper_CUDA__native_layer_norm_backward(at::Tensor const&, at::Tensor const&, c10::ArrayRef<c10::SymInt>, at::Tensor const&, at::Tensor const&, std::optional<at::Tensor> const&, std::optional<at::Tensor> const&, std::array<bool, 3ul>) from RegisterCUDA_0.cpp:0 ``` Now we only call `sum(0)` on tensors that are defined and properly guard the `sum(0)` and assignment. Pull Request resolved: pytorch#156600 Approved by: https://github.com/eqy, https://github.com/ngimel

For tensor with non-zero offset, it must be multiplied by element size Add regression test by creating Tensor in array of 6 elements with offset 3, which before the fix crashed with ``` C++ exception with description "setStorage: sizes [3, 3], strides [0, 1], storage offset 3, and itemsize 4 requiring a storage size of 24 are out of bounds for storage of size 15 Exception raised from checkInBoundsForStorage at /Users/nshulga/git/pytorch/pytorch/aten/src/ATen/native/Resize.h:123 (most recent call first): frame #0: c10::Error::Error(c10::SourceLocation, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>) + 56 (0x104a9cd44 in libc10.dylib) frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) + 120 (0x104a9a05c in libc10.dylib) frame #2: void at::native::checkInBoundsForStorage<long long>(c10::ArrayRef<long long>, c10::ArrayRef<long long>, long long, caffe2::TypeMeta const&, c10::Storage const&) + 656 (0x111dbd314 in libtorch_cpu.dylib) frame #3: void at::native::setStrided<long long>(at::Tensor const&, c10::ArrayRef<long long>, c10::ArrayRef<long long>, long long) + 152 (0x111dcd22c in libtorch_cpu.dylib) frame #4: at::native::as_strided_tensorimpl(at::Tensor const&, c10::ArrayRef<long long>, c10::ArrayRef<long long>, std::__1::optional<long long>) + 312 (0x111dccf98 in libtorch_cpu.dylib) frame #5: c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (at::Tensor const&, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, std::__1::optional<c10::SymInt>), &at::(anonymous namespace)::(anonymous namespace)::wrapper_CPU__as_strided(at::Tensor const&, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, std::__1::optional<c10::SymInt>)>, at::Tensor, c10::guts::typelist::typelist<at::Tensor const&, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, std::__1::optional<c10::SymInt>>>, at::Tensor (at::Tensor const&, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, std::__1::optional<c10::SymInt>)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, std::__1::optional<c10::SymInt>) + 104 (0x1129a1e94 in libtorch_cpu.dylib) frame #6: at::_ops::as_strided::call(at::Tensor const&, c10::ArrayRef<c10::SymInt>, c10::ArrayRef<c10::SymInt>, std::__1::optional<c10::SymInt>) + 476 (0x112200ad0 in libtorch_cpu.dylib) frame #7: at::Tensor::as_strided(c10::ArrayRef<long long>, c10::ArrayRef<long long>, std::__1::optional<long long>) const + 236 (0x1115db098 in libtorch_cpu.dylib) frame #8: at::native::expand(at::Tensor const&, c10::ArrayRef<long long>, bool) + 348 (0x111dcc0d4 in libtorch_cpu.dylib) frame #9: c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (c10::DispatchKeySet, at::Tensor const&, c10::ArrayRef<c10::SymInt>, bool), &torch::ADInplaceOrView::(anonymous namespace)::expand(c10::DispatchKeySet, at::Tensor const&, c10::ArrayRef<c10::SymInt>, bool)>, at::Tensor, c10::guts::typelist::typelist<c10::DispatchKeySet, at::Tensor const&, c10::ArrayRef<c10::SymInt>, bool>>, at::Tensor (c10::DispatchKeySet, at::Tensor const&, c10::ArrayRef<c10::SymInt>, bool)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, c10::ArrayRef<c10::SymInt>, bool) + 116 (0x1157ac410 in libtorch_cpu.dylib) frame #10: c10::impl::wrap_kernel_functor_unboxed_<c10::impl::detail::WrapFunctionIntoFunctor_<c10::CompileTimeFunctionPointer<at::Tensor (c10::DispatchKeySet, at::Tensor const&, c10::ArrayRef<c10::SymInt>, bool), &torch::autograd::VariableType::(anonymous namespace)::expand(c10::DispatchKeySet, at::Tensor const&, c10::ArrayRef<c10::SymInt>, bool)>, at::Tensor, c10::guts::typelist::typelist<c10::DispatchKeySet, at::Tensor const&, c10::ArrayRef<c10::SymInt>, bool>>, at::Tensor (c10::DispatchKeySet, at::Tensor const&, c10::ArrayRef<c10::SymInt>, bool)>::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&, c10::ArrayRef<c10::SymInt>, bool) + 992 (0x114e8b010 in libtorch_cpu.dylib) frame #11: at::_ops::expand::call(at::Tensor const&, c10::ArrayRef<c10::SymInt>, bool) + 316 (0x112743c90 in libtorch_cpu.dylib) frame #12: at::expand_size(at::Tensor const&, c10::ArrayRef<long long>) + 164 (0x1047d82b4 in basic) frame #13: BasicTest_TestForBlobResizeCPU_Test::TestBody() + 284 (0x1047d8048 in basic) ``` Pull Request resolved: pytorch#158690 Approved by: https://github.com/angelayi

) Summary: This diff fixes two things which come up when testing a tgif-published pt2 model remote net: 1) Updates isSameDevice to handle meta device to avoid this error: ``` what(): Unsupported device typemeta and meta Exception raised from isSameDevice at fbcode/caffe2/torch/nativert/executor/PlacementUtils.cpp:20 ``` 2. Updates xl weight v2 loading logic in Weights.cpp to handle non-TBE xl-weights. Today, we enforce the device is the same for an old weight and new weight when replacing with ModelRunnerAdapter.setAttr(). However, the way we replace non-TBE xl weights is to find any weights on "meta" device and then replace them with their correct weight with real device from xl_weights folder. Therefore, the new weight and old weight will always have different devices and the device check is invalid. I don't think we've run into this so far bc non-TBE xl weights have not been thoroughly tested until now. Test Plan: Run MRS you model merge net, which uses non-TBE xl weights. Confirm that before change #1 we get error: ``` Unsupported device typemeta and meta ``` Then after change #1 and before change #2 we get: ``` what(): Mismatched device for merge.user_tower.linear.weight: meta vs cpu Exception raised from validateValue at fbcode/caffe2/torch/nativert/executor/Weights.cpp:374 ``` After change run is successful Command: ``` MODEL_ENTITY_ID=921242082 SNAPSHOT_ID=1269 module_name=merge SAMPLE_INPUT_DIR=/data/users/georgiaphillips/models/921242082/${SNAPSHOT_ID}/${module_name}_archive/package/data/sample_inputs buck2 run mode/dev-nosan -c fbcode.nvcc_arch=h100,a100 -c fbcode.enable_gpu_sections=true caffe2/torch/fb/model_transform/fx2trt/packaging:load_net_predictor -- --loadMode=Benchmark --inputNetFile=/data/users/$USER/models/${MODEL_ENTITY_ID}/${SNAPSHOT_ID}/${MODEL_ENTITY_ID}_${SNAPSHOT_ID}.predictor.${module_name} --moduleName=${module_name} --submodToDevice="merge|cuda0" --benchmarkEnableProfiling=false --disableStaticRuntime=true --doNotRandomizeSampleInputs=true --benchmarkDontRebatchSamples=true --pytorch_predictor_sigmoid_static_dispatch_enable=false --pytorch_predictor_sigmoid_graph_passes_enable=false --sampleInputFilePath=${SAMPLE_INPUT_DIR}/${module_name}.pt ``` Rollback Plan: Differential Revision: D80713052 Pull Request resolved: pytorch#162842 Approved by: https://github.com/henryoier

…rch#165479) These happen when building with CMAKE_BUILD_TYPE=RelWithAssert This should fix two types of failures that started with pytorch#163665 Disclaimer that I used a lot of AI since I don't how pybind works or what refcounts and pointers are, so idk if this is a good solution, or even a solution at all (fwiw the tests pass now) The first one type is Truncated: ``` default_pg, _ = _new_process_group_helper( File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2096, in _new_process_group_helper backend_class = creator_fn(dist_backend_opts, backend_options) File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/distributed/fake_pg.py", line 25, in _create_fake_pg return FakeProcessGroup._create_internal( RuntimeError: new_refcount != 1 INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/c10/util/intrusive_ptr.h":319, please report a bug to PyTorch. intrusive_ptr: Cannot increase refcount after it reached zero. Exception raised from retain_ at /var/lib/jenkins/workspace/c10/util/intrusive_ptr.h:319 (most recent call first): C++ CapturedTraceback: #4 std::_Function_handler<std::shared_ptr<c10::LazyValue<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > const> (), c10::SetStackTraceFetcher(std::function<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > ()>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) from Logging.cpp:0 #5 c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) from ??:0 #6 c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) from ??:0 #7 c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, char const*) from ??:0 #8 void pybind11::class_<c10d::FakeProcessGroup, (anonymous namespace)::IntrusivePtrNoGilDestructor<c10d::FakeProcessGroup> >::init_instance<(anonymous namespace)::IntrusivePtrNoGilDestructor<c10d::FakeProcessGroup>, 0>(pybind11::detail::instance*, void const*) from init.cpp:0 #9 pybind11::detail::type_caster_generic::cast(void const*, pybind11::return_value_policy, pybind11::handle, pybind11::detail::type_info const*, void* (*)(void const*), void* (*)(void const*), void const*) from :0 #10 pybind11::cpp_function::initialize<torch::distributed::c10d::(anonymous namespace)::c10d_init(_object*, _object*)::{lambda(int, int, c10::intrusive_ptr<c10d::FakeProcessGroup::Options, c10::detail::intrusive_target_default_null_type<c10d::FakeProcessGroup::Options> >)pytorch#127}, c10::intrusive_ptr<c10d::FakeProcessGroup, c10::detail::intrusive_target_default_null_type<c10d::FakeProcessGroup> >, int, int, c10::intrusive_ptr<c10d::FakeProcessGroup::Options, c10::detail::intrusive_target_default_null_type<c10d::FakeProcessGroup::Options> >, pybind11::name, pybind11::scope, pybind11::sibling, pybind11::arg, pybind11::arg, pybind11::arg_v>(torch::distributed::c10d::(anonymous namespace)::c10d_init(_object*, _object*)::{lambda(int, int, c10::intrusive_ptr<c10d::FakeProcessGroup::Options, c10::detail::intrusive_target_default_null_type<c10d::FakeProcessGroup::Options> >)pytorch#127}&&, c10::intrusive_ptr<c10d::FakeProcessGroup, c10::detail::intrusive_target_default_null_type<c10d::FakeProcessGroup> > (*)(int, int, c10::intrusive_ptr<c10d::FakeProcessGroup::Options, c10::detail::intrusive_target_default_null_type<c10d::FakeProcessGroup::Options> >), pybind11::name const&, pybind11::scope const&, pybind11::sibling const&, pybind11::arg const&, pybind11::arg const&, pybind11::arg_v const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) from init.cpp:0 ``` and I fix it here by getting rid of `DontIncreaseRefcount` and using make_intrusive to do the ref count handling instead. However, I also had to move the constructor to be public, which I think is not good, based on the reasoning of the original PR The other one type is ``` Traceback (most recent call last): File "/var/lib/jenkins/workspace/test/test_testing.py", line 2415, in test_no_warning_on_import self.assertEqual(out, "") File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 4233, in assertEqual raise error_metas.pop()[0].to_error( # type: ignore[index] AssertionError: String comparison failed: "/opt/conda/envs/py_3.10/lib/python3.10/s[352 chars]):\n" != '' - /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/distributed/__init__.py:29: FutureWarning: pybind11-bound class 'torch._C._distributed_c10d.FakeProcessGroup' is using an old-style placement-new '__init__' which has been deprecated. See the upgrade guide in pybind11's docs. This message is only visible when compiled in debug mode. - if is_available() and not torch._C._c10d_init(): To execute this test, run the following from the base repo dir: python test/test_testing.py TestImports.test_no_warning_on_import ``` which I fix by getting rid of the `__init__` which I think is ok since it'll just error if you try to make one? Pull Request resolved: pytorch#165479 Approved by: https://github.com/ezyang

…torch#168129) (This PR will be rebased on pytorch#166174) (There are other PR which updates NCCL version: pytorch#168091) We did the following thing: 1. To add exchange of buffer ptr and signal pad ptr via NCCL device API introduced in nccl 2.28. 2. With #1, we showed that the symmem from nccl backend works with existing one_shot_all_reduce kernel (Add a UT for it) 3. Add a simple put, put with signal, wait for signal and get. So that symmem's one side API works. 4. Show that symmem from nccl backend works with traditional c10d collective as well in UT. 5. Stored DevComm inside symmetric memory so that users can access to it for customized kernel. Resolves pytorch#167682 Pull Request resolved: pytorch#168129 Approved by: https://github.com/kwen2501, https://github.com/ngimel, https://github.com/atalman

If another static object (like `g_device_config_parse_hook_registry_instance` created by the `REGISTER_ALLOCATOR_CONFIG_PARSE_HOOK` macro) tries to call `registerDeviceConfigParserHook` before `device_config_parser_hook_` is initialized, assigning to it (operator=) can fail, which leads to a runtime error. When I use a compilation optimization of ` -O1` I see this issue: ``` [src/libcxx/include/__functional/function.h:496]:14: runtime error: member access within null pointer of type 'const __policy' #0 0x563224e28b78 in operator= [crosstool/v18/stable/src/libcxx/include/__functional/function.h:496]:14 #1 0x563224e28b78 in operator= [crosstool/v18/stable/src/libcxx/include/__functional/function.h:483]:19 #2 0x563224e28b78 in operator= [crosstool/v18/stable/src/libcxx/include/__functional/function.h:727]:8 #3 0x563224e28b78 in c10::CachingAllocator::AcceleratorAllocatorConfig::registerDeviceConfigParserHook(std::__u::function<void (std::__u::basic_string<char, std::__u::char_traits<char>, std::__u::allocator<char>> const&)>&&, std::__u::unordered_set<std::__u::basic_string<char, std::__u::char_traits<char>, std::__u::allocator<char>>, std::__u::hash<std::__u::basic_string<char, std::__u::char_traits<char>, std::__u::allocator<char>>>, std::__u::equal_to<std::__u::basic_string<char, std::__u::char_traits<char>, std::__u::allocator<char>>>, std::__u::allocator<std::__u::basic_string<char, std::__u::char_traits<char>, std::__u::allocator<char>>>> const&) [torch/c10/core/AllocatorConfig.h:263]:32 #4 0x563224e28e9d in DeviceConfigParserHookRegistry [torch/c10/core/AllocatorConfig.h:369]:5 #5 0x563224e28e9d in __cxx_global_var_init.34 [torch/c10/cuda/CUDAAllocatorConfig.cpp:195]:1 #6 0x563224e28e9d in _GLOBAL__sub_I_CUDAAllocatorConfig.cpp torch/c10/cuda/CUDAAllocatorConfig.cpp #7 0x5632459709ac in __libc_csu_init /[usr/grte/v5/debug-src/src/csu/elf-init.c:88]:7 #8 0x7f748b9562e7 in __libc_start_main (/usr/grte/v5/lib64/libc.so.6+0x612e7) (BuildId: ca23ec6d935352118622ce674a8bb52d) #9 0x5632018f3729 in _start /usr/grte/v5/debug-src/src/csu/../sysdeps/x86_64/start.S:120 ``` Pull Request resolved: pytorch#172581 Approved by: https://github.com/guangyey, https://github.com/albanD

…ytorch#174247) Summary: This patch fixes the loss of signal info in Coredumps produced by caffe2 apps when they crash. The culprit is the signal handler's call to `raise` after unregistering itself. Raise under the hood actually calls `tgkill` which replaces whatever the data into the `siginfo_t` with the uid and pid of the calling process. This means when the signal and re-raised and the process coredumps, the reason for the coredump is something like `SEGV sent by=your pid, your user` without the address info or the SI_CODE from the original signal. We fix this by calling raise signal directly with the original signal. This is a port of yfeldblum's change in [Folly Signal Handler](facebook/folly@79d7f8e) to caffe2. Test Plan: In the diff above this one creates a small app that loads the caffe2 app and then SEGV's. Then inspecting the core locally ``` (lldb) thread siginfo thread #1: tid = 1711969, 0x000000000024f76a, name = 'signal_handler_', stop reason = SIGSEGV: address not mapped to object (fault address=0x1000) (__lldb_siginfo_t) __lldb_siginfo = { si_signo = 11 si_errno = 0 si_code = 1 __pad0 = 0 _sifields = { _kill = (si_pid = 4096, si_uid = 0) _timer = { si_tid = 4096 si_overrun = 0 si_sigval = (sival_int = 0, sival_ptr = 0x0000000000000000) } _rt = { si_pid = 4096 si_uid = 0 si_sigval = (sival_int = 0, sival_ptr = 0x0000000000000000) } _sigchld = (si_pid = 4096, si_uid = 0, si_status = 0, si_utime = 0, si_stime = 0) _sigfault = { si_addr = 0x0000000000001000 si_addr_lsb = 0 _bounds = { _addr_bnd = (_lower = 0x0000000000000000, _upper = 0x0000000000000000) _pkey = 0 } } _sigpoll = (si_band = 4096, si_fd = 0) _sigsys = (_call_addr = 0x0000000000001000, _syscall = 0, _arch = 0) } } ``` And we see the siginfo contains the address which triggered the original SEGV. Differential Revision: D92093984 Pull Request resolved: pytorch#174247 Approved by: https://github.com/Skylion007

…c8 kernel (pytorch#174362) This will allow `sm_103` devices call vec8 kernels. Verification script: ```Python import torch from torch.profiler import profile, ProfilerActivity device = torch.device("cuda") for dtype in (torch.bfloat16, torch.float16,): x = torch.randn(1024, device=device, dtype=dtype) with profile(activities=[ProfilerActivity.CUDA], record_shapes=True) as prof: y = torch.relu(x) stats = prof.key_averages() for entry in stats: if "at::native::vectorized_elementwise_kernel" in entry.key: print(entry.key) ``` Before: ``` void at::native::vectorized_elementwise_kernel<4, at::native::(anonymous namespace)::launch_clamp_scalar(at::TensorIteratorBase&, c10::Scalar, c10::Scalar, at::native::detail::ClampLimits)::{lambda()#1}::operator()() const::{lambda()#9}::operator()() const::{lambda(c10::BFloat16)#1}, std::array<char*, 2ul> >(int, at::native::(anonymous namespace)::launch_clamp_scalar(at::TensorIteratorBase&, c10::Scalar, c10::Scalar, at::native::detail::ClampLimits)::{lambda()#1}::operator()() const::{lambda()#9}::operator()() const::{lambda(c10::BFloat16)#1}, std::array<char*, 2ul>) void at::native::vectorized_elementwise_kernel<4, at::native::(anonymous namespace)::launch_clamp_scalar(at::TensorIteratorBase&, c10::Scalar, c10::Scalar, at::native::detail::ClampLimits)::{lambda()#1}::operator()() const::{lambda()#8}::operator()() const::{lambda(c10::Half)#1}, std::array<char*, 2ul> >(int, at::native::(anonymous namespace)::launch_clamp_scalar(at::TensorIteratorBase&, c10::Scalar, c10::Scalar, at::native::detail::ClampLimits)::{lambda()#1}::operator()() const::{lambda()#8}::operator()() const::{lambda(c10::Half)#1}, std::array<char*, 2ul>) ``` After: ``` void at::native::vectorized_elementwise_kernel<8, at::native::(anonymous namespace)::launch_clamp_scalar(at::TensorIteratorBase&, c10::Scalar, c10::Scalar, at::native::detail::ClampLimits)::{lambda()#1}::operator()() const::{lambda()#9}::operator()() const::{lambda(c10::BFloat16)#1}, std::array<char*, 2ul> >(int, at::native::(anonymous namespace)::launch_clamp_scalar(at::TensorIteratorBase&, c10::Scalar, c10::Scalar, at::native::detail::ClampLimits)::{lambda()#1}::operator()() const::{lambda()#9}::operator()() const::{lambda(c10::BFloat16)#1}, std::array<char*, 2ul>) void at::native::vectorized_elementwise_kernel<8, at::native::(anonymous namespace)::launch_clamp_scalar(at::TensorIteratorBase&, c10::Scalar, c10::Scalar, at::native::detail::ClampLimits)::{lambda()#1}::operator()() const::{lambda()#8}::operator()() const::{lambda(c10::Half)#1}, std::array<char*, 2ul> >(int, at::native::(anonymous namespace)::launch_clamp_scalar(at::TensorIteratorBase&, c10::Scalar, c10::Scalar, at::native::detail::ClampLimits)::{lambda()#1}::operator()() const::{lambda()#8}::operator()() const::{lambda(c10::Half)#1}, std::array<char*, 2ul>) ``` Pull Request resolved: pytorch#174362 Approved by: https://github.com/ngimel

…nces between x86 vs aarch64 (pytorch#176085) In the test: ``` python test/cpp_extensions/test_libtorch_agnostic.py TestLibtorchAgnosticCUDA.test_std_cuda_check_error_show_cpp_stacktraces_True_cuda ``` it raises an exception when calling `STD_CUDA_CHECK(cudaSetDevice(99999));` which got the expected `CUDA error: invalid device` message. However, the expected string for the C++ stack trace is different between `x86` vs `aarch64` due perhaps in these issues: - pytorch#119905 - pytorch#134387 In the current setup when getting a stack trace string: - x86 contains `C++ CapturedTraceback:` - aarch64 contains `Exception raised from` + `frame #` An example of the full string from an aarch64 system when : ``` AssertionError: 'C++ CapturedTraceback:' not found in 'CUDA error: invalid device ordinal\nGPU device may be out of range, do you have enough GPUs?\nCUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n\nException raised from test_std_cuda_check_error at /opt/pytorch/pytorch/test/cpp_extensions/libtorch_agn_2_10_extension/csrc/test_std_cuda_check.cu:23 (most recent call first):\nframe #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0xd4 (0xe471ebcd39f4 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10.so)\nframe #1: <unknown function> + 0x43f998 (0xe471ebdcf998 in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10_cuda.so)\nframe #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, unsigned int, bool) + 0x1bc (0xe471ebdcfc0c in /usr/local/lib/python3.12/dist-packages/torch/lib/libc10_cuda.so)\nframe #3: torch_c10_cuda_check_msg + 0x1c (0xe471ef335c4c in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cuda.so)\nframe #4: test_std_cuda_check_error() + 0x58 (0xe470cd396678 in /opt/pytorch/pytorch/test/cpp_extensions/libtorch_agn_2_10_extension/install/usr/local/lib/python3.12/dist-packages/libtorch_agn_2_10/_C.so)\nframe #5: c10::BoxedKernel::makeFromFunctor<StableIValueBoxedKernel>(std::unique_ptr<StableIValueBoxedKernel, std::default_delete<StableIValueBoxedKernel> >)::{lambda(c10::OperatorKernel*, c10::OperatorHandle const&, c10::DispatchKeySet, std::vector<c10::IValue, std::allocator<c10::IValue> >*)#1}::_FUN(c10::OperatorKernel*, c10::OperatorHandle const&, c10::DispatchKeySet, std::vector<c10::IValue, std::allocator<c10::IValue> >*) + 0x16c (0xe47211cd419c in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)\nframe #6: <unknown function> + 0x61d34bc (0xe47211cf34bc in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so)\nframe #7: <unknown function> + 0xe6c324 (0xe4721532c324 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so)\nframe #8: <unknown function> + 0xe6c7e0 (0xe4721532c7e0 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so)\nframe #9: <unknown function> + 0xd3907c (0xe472151f907c in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so)\nframe #10: <unknown function> + 0x5ccbf8 (0xe47214a8cbf8 in /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_python.so)\nframe #11: /usr/bin/python() [0x504a34]\nframe #12: PyObject_Call + 0x6c (0x4c633c in /usr/bin/python)\nframe #13: _PyEval_EvalFrameDefault + 0x3ea0 (0x568564 in /usr/bin/python)\nframe #14: _PyObject_Call_Prepend + 0xc4 (0x4c5934 in /usr/bin/python)\nframe #15: /usr/bin/python() [0x52a070]\nframe #16: _PyObject_MakeTpCall + 0x78 (0x4c3e58 in /usr/bin/python)\nframe #17: _PyEval_EvalFrameDefault + 0x8a0 (0x564f64 in /usr/bin/python)\nframe #18: PyEval_EvalCode + 0x130 (0x5632b4 in /usr/bin/python)\nframe #19: PyRun_StringFlags + 0xe0 (0x59c330 in /usr/bin/python)\nframe #20: PyRun_SimpleStringFlags + 0x44 (0x67ebc4 in /usr/bin/python)\nframe #21: Py_RunMain + 0x390 (0x68b380 in /usr/bin/python)\nframe #22: Py_BytesMain + 0x28 (0x68ae88 in /usr/bin/python)\nframe #23: <unknown function> + 0x284c4 (0xe47216b084c4 in /lib/aarch64-linux-gnu/libc.so.6)\nframe #24: __libc_start_main + 0x98 (0xe47216b08598 in /lib/aarch64-linux-gnu/libc.so.6)\nframe #25: _start + 0x30 (0x5f6770 in /usr/bin/python)\n\n' To execute this test, run the following from the base repo dir: python test/cpp_extensions/test_libtorch_agnostic.py TestLibtorchAgnosticCUDA.test_std_cuda_check_error_show_cpp_stacktraces_True_cuda ``` Pull Request resolved: pytorch#176085 Approved by: https://github.com/eqy

## Summary Add `module_error_inputs_torch_nn_Embedding` function to test error messages for invalid inputs to `nn.Embedding` module. ## Motivation Currently, `torch.nn.Embedding` does not have `module_error_inputs_func` defined in `common_modules.py`. This PR adds error input tests to enable regression testing for error messages and follow the pattern already established for other modules (BatchNorm, GroupNorm, RNN cells, etc.). ## Test Cases Added 1. **Out of range indices**: Tests IndexError when index exceeds `num_embeddings` - Input: `Embedding(10, 3)` with indices `[0, 5, 15]` (15 is out of range) - Expected: `IndexError: index out of range in self` - **Note**: Only tested on CPU - CUDA triggers kernel assertion instead of Python exception 2. **Float indices**: Tests RuntimeError when float tensor used instead of Long/Int - Input: `Embedding(10, 3)` with float tensor `[1.5, 2.5]` - Expected: `RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int` 3. **Negative num_embeddings**: Tests RuntimeError when constructor has negative dimension - Input: `Embedding(-1, 3)` - Expected: `RuntimeError: Trying to create tensor with negative dimension` ## Test Environment - Tested on H200 GPU with CUDA 12.8 - Verified error messages match on CPU - CUDA tests pass for dtype and construction errors Fixes pytorch#174179 Pull Request resolved: pytorch#174180 Approved by: https://github.com/albanD

…process group (pytorch#192110) Summary: ProcessGroup::splitGroup did auto backendOpts = opts.has_value() ? *opts : parentBackend->getBackendOptions(); backendOpts->group_name = groupName; backendOpts->timeout = ...; backendOpts->group_desc = groupDesc; auto splitBackend = parentBackend->split(store, ranks, backendOpts); and every getBackendOptions() in tree returns the backend's live options_, not a copy -- ProcessGroupGloo, ProcessGroupNCCL, nccl2 and FakeProcessGroup all just cast options_, and ProcessGroupWrapper/LazyBackend delegate. So a split rewrites the *parent's* group_name, timeout and group_desc, and the child ends up holding the very same Options object. ProcessGroupGloo::split then writes `glooOpts->global_ranks_in_group = std::move(globalRanksInGroup)` onto it, i.e. the parent's rank map is replaced by the child's, and the parent's next split indexes a vector sized for the child: parent BEFORE any split: group_name='0' timeout=0:01:51 ranks=[] parent AFTER split #1: group_name='0:split:[0, 1]' timeout=0:03:42 ranks=[0, 1] child1.options IS parent.options: True [rank2] split #2 child ranks = [139894703605936, 1900999358] (expected [0, 2]) [rank0] split #2 child ranks = [0, 1954417006] (expected [0, 2]) That is a heap read past the end, on a stock 4-rank CPU-only gloo world driven entirely through the public ProcessGroup.split_group(ranks) pybind, whose opts argument defaults to nullopt (test/distributed/test_device_mesh.py already calls it that way). The child it produces has garbage global ranks, so its FlightRecorder identity and any rank translation done through it are wrong; on a different heap layout the same read is a SIGSEGV. mergeRemoteGroup has the same aliasing and no escape hatch at all: it takes no opts parameter, so it always rewrites the parent's options and hands the live object to merge(), leaving parent and merged child permanently sharing one Options for the rest of the process. All four defects fixed here are stock. None needs nccl2, or any custom backend: the aliasing is a 4-rank CPU-only gloo world, the Python options substitution is any "cpu:gloo,cuda:nccl" world, the split-once-per-backend hang is a bare "gloo" world on an accelerator host, and the deepcopy TypeError is any gloo-backed group. What the nccl2 migration changed is only reachability: it routes CPU coordination groups through split_group (see MIGRATION_RULES.md), which is what made every rank start printing the ProcessGroupGloo::split options warning on every compound split. That warning is the thread this change was pulled from. The order of discovery matters for reading the rest of this message. The warning looked cosmetic, so the obvious Python one-liner -- stop substituting pg_options, let the C++ per-backend fallback do its job -- was implemented and probed. It removed the warning and gave the gloo child the right group_name, timeout and ranks, and it turned the cosmetic problem into a memory-safety one, because the fallback path is exactly the aliasing above. That is how the C++ defect was found. The split-once-per-device-type hang was then found while verifying the Python change, because deleting the deepcopy is what first makes dist.split_group reachable at all on a gloo world; and the gloo _Options TypeError surfaced in the same runs. Fix it in C++, at the point where the options are handed to a backend: add a virtual Backend::Options::clone() and copy per backend in splitGroup and mergeRemoteGroup. Threading a per-device opts map through splitGroup instead was considered and is strictly worse -- it does not fix the no-entry fallback (a device with no map entry still lands on the parent's live options), does not fix mergeRemoteGroup at all, changes the signature and every caller, and the right per-device options are simply the parent's own, which clone() gives for free. Making each backend's split()/merge() defensively copy what it is handed was also rejected: the parent's group_name, timeout and group_desc are already overwritten before split() is ever called, so the copy has to happen at the call site or the parent is corrupted regardless. And copying in Python -- which is what the deleted deepcopy was -- requires every Options subclass to carry a pickling pybind, which is precisely the requirement gloo failed. clone() is deliberately not pure. ProcessGroupXCCL::Options derives from Backend::Options but lives out of tree in third_party/torch-xpu-ops, fetched by caffe2/CMakeLists.txt and pinned by third_party/xpu.txt, so a pure virtual would make it abstract and break the XPU build. For the same reason the call site guards against slicing: auto copy = opts->clone(); if (copy == nullptr || typeid(*copy) != typeid(*opts)) { return opts; // subclass has not adopted clone(): keep legacy sharing } so a backend that has not adopted clone() keeps exactly today's behaviour rather than being handed a sliced base Options -- aliasing is a bug, but handing a backend an object of the wrong dynamic type is worse, and the backend cannot detect it. RTTI is already required by c10d (the dynamic_intrusive_pointer_casts in ProcessGroupGloo.cpp and FakeProcessGroup.hpp), so the typeid test costs nothing new. An explicit caller-supplied opts is now applied only to the group's *default* backend type; the other device legs inherit a clone of their own backend's options. That mirrors what pg_options already means for init_process_group, where _new_process_group_helper hands it to the backend creator that understands it, and it is what stops a ProcessGroupNCCL::Options from reaching the gloo leg of a "cpu:gloo,cuda:nccl" group. The caller's object is cloned too, so split_group no longer mutates it. While here, split (and merge) each backend *instance* once rather than once per device type. splitGroup iterates deviceTypeToBackendType_, but several device types can map to the same backend object: init_process_group("gloo") registers one ProcessGroupGloo for both cpu and cuda, and ProcessGroup::setBackend explicitly reuses the backend already registered for a backend type. The loop therefore built a second child that was immediately discarded -- after it had already rendezvoused on the *same* PrefixStore("<groupName>/") keys as the real child. The two gloo contexts race for "<group>/0/rank_N" and peers pair with the discarded group: $ python bug3b_double_split_same_backend.py gloo 5 # cpu+cuda -> one PG [rank1] split 0 done <hang; rank0 never returns from split 0> $ python bug3b_double_split_same_backend.py cpu:gloo 5 # one device type [rank0] split 0..4 done / [rank1] split 0..4 done This is pre-existing and independent, but it blocks the change below: removing the Python deepcopy is what makes dist.split_group reachable at all on a gloo world, and the first thing it then hits is this hang. On the Python side, split_group substituted `pg_options = copy.deepcopy(parent_backend.options)` where parent_backend is the *accelerator* backend, and passed that one object to every device's split(). On a "cpu:gloo,cuda:nccl" world the gloo leg got a ProcessGroupNCCL::Options, warned "Tried to pass options to ProcessGroupGloo::split that are not ProcessGroupGloo::Options. Falling back to default options.", and silently lost the caller's timeout and group_name (123 s and a real name became 0:30:00 and ''; the empty name is what the ctor feeds to setGroupUid, so the child's pg uid was empty too). Not cosmetic even for backends that only warn -- a 30-minute timeout where the caller asked for 123 s changes when a hang is detected -- and a backend that is strict about its Options type hard-errors on the same mismatch instead of warning. Worse, on a *gloo* world the accelerator backend is the gloo backend, and ProcessGroupGloo::_Options has no __copy__/__deepcopy__ pybind (unlike the NCCL, nccl2 and fake Options), so dist.split_group did not merely warn, it threw: TypeError: cannot pickle 'torch._C._distributed_c10d._Options' object i.e. dist.split_group was unusable for every gloo-backed group on an accelerator host, in stock, with no migration involved -- nothing in stock CI splits a bare gloo world on such a host. Delete the deepcopy and leave pg_options as the caller passed it; the C++ side now gives each device backend a copy of its own options, and the pickling requirement disappears with the deepcopy that imposed it. The two halves must land together. Leaving pg_options=None *without* the C++ clone was tried and is a memory-safety regression, not a fix: it routes the gloo leg straight to parentBackend->getBackendOptions(), i.e. exactly the aliasing above, so the parent's global_ranks_in_group is overwritten with the child's and the parent's next split reads out of bounds. The Python-only "one-line fix" is not viable on its own. There is no downstream fix for any of this. A caller can pass explicit pg_options, but there is no per-device options API to pass, so on a multi-device world one of the legs is still wrong; merge_remote_group takes no opts at all; and nothing outside c10d can stop splitGroup writing group_name, timeout and global_ranks_in_group onto the parent's live object. The only downstream "mitigation" is to never split the same parent twice -- which DeviceMesh, vLLM and our own migration all do routinely. While here, close the netName leak this change would otherwise multiply. nccl2's Options::clone(), added above, is c10::intrusive_ptr<::c10d::Backend::Options> clone() const override { auto copy = c10::make_intrusive<Options>(*this); copy->config = cloneNcclConfig(config); return copy; } and cloneNcclConfig strdup's config.netName while ncclConfig_t records no owner. splitGroup and mergeRemoteGroup call cloneOptions() once per backend per split/merge, into a backendOpts local that nccl2's split() never retains -- it builds its own childOpts -- so that strdup is unreachable the moment the loop iteration ends. The allocation itself is not new: cloneNcclConfig and its use in nccl2's split() both predate this stack, so the once-per-process leak is inherited. What is ours is the *rate*: cloning per split/merge turns it into one leaked allocation per split_group / merge_remote_group per PG per rank whenever a caller sets net_name. We did not create the leak, we multiplied it, so the ownership fix belongs in this commit rather than stacked on top of it. clone() now ties the allocation it makes to the Options that made it: copy->config = cloneNcclConfig(config); if (copy->config.netName != nullptr) { copy->owned_net_name_ = std::shared_ptr<const char>( copy->config.netName, [](const char* p) { std::free(const_cast<char*>(p)); }); } with a private `std::shared_ptr<const char> owned_net_name_` member. The ownership model, stated across construct / clone / destroy: - Options(bool): netName is NCCL_CONFIG_UNDEF_PTR (NULL). Owns nothing. - Options(const BaseOptions&): shares the base's netName, and the base keeps owning it -- untracked, unchanged, unfreed. A conversion that deep-copied here would start freeing a string a live stock Options still points at. - the copy constructor and copy assignment stay `= default`, so the shared_ptr is copied along with every other field: copies share the allocation and the reference count, and the string outlives every Options pointing at it and is freed exactly once, when the last of them dies. - clone() is the sole place ownership is established, and only for the one allocation it makes. - ~Options() stays `= default`; the member does the freeing, so there is no hand-written destructor to keep in sync as fields are added. - anything else that lands in config.netName -- the NCCLConfig pybind's strdup, a config assigned wholesale from Python, the clone split() hands to ncclCommSplit -- is untracked and untouched, exactly as before. If Python replaces config on a cloned Options, the shared_ptr still frees our now unreferenced buffer and leaves theirs alone. Ownership is deliberately taken *only* for an allocation an Options makes for itself and never hands to NCCL. The pinned third_party/nccl (v2.29.7-1) copies the caller's string -- src/init.cc:1863-1865 malloc's strlen(tmpNetName)+1 and memcpy's into comm->config.netName, and src/init.cc:349 frees only that copy -- so it never frees the caller's pointer, and everything cloneNcclConfig allocates leaks under this tree. But the comment already at nccl2/ProcessGroupNCCL.hpp:52-56 records that *some* NCCL versions free the caller's string on comm destruction, PyTorch can be built against a system NCCL, and the cutoff version could not be established (third_party/nccl carries one squashed release commit and no history). A destructor that freed whatever is in config.netName would therefore double-free on any version matching that comment, since the same pointer is handed to ncclCommInitRankConfig/ncclCommSplit, and would dangle a live stock Options in the base-adopting case. So the strdup's that split() and ProcessGroupNCCLLazy hand to NCCL are left alone; fixing those means either pinning a version cutoff or strdup'ing separately at every NCCL handoff, which only moves the leak. The `= default` special members are load-bearing and should not be tidied into hand-written ones. An intermediate version of this did exactly that -- a copy constructor that strdup'ed plus a destructor that free'd a raw char* member -- and it compiled, leaked nothing, and silently stopped copying abort_process_on_timeout_or_error, because a hand-written copy constructor initialises the base and then leaves *derived* members at their default initialisers. ProcessGroupNCCL2Test::test_options_type caught it, as "AssertionError: True is not false" through copy.deepcopy. The shared_ptr shape has no such trap: copy semantics stay compiler-generated, and therefore stay correct for fields added later. Test Plan: The four repro scripts, re-run against the rebuilt tree. bug3_gloo_split_alias.py (4 ranks, CPU only) now keeps the parent untouched, reports that the child's options are a distinct object, and gives all four ranks the correct second-split ranks: python triage/bug3_gloo_split_alias.py 29541 parent BEFORE any split: group_name='0' timeout=0:01:51 ranks=[] parent AFTER split #1: group_name='0' timeout=0:01:51 ranks=[] child #1: group_name='0:split:[0, 1]' timeout=0:03:42 ranks=[0, 1] child1.options IS parent.options: False [rank0] split #2 child global_ranks_in_group = [0, 2] (expected [0, 2]) OK [rank1] split #2 child global_ranks_in_group = [1, 3] (expected [1, 3]) OK [rank2] split #2 child global_ranks_in_group = [0, 2] (expected [0, 2]) OK [rank3] split #2 child global_ranks_in_group = [1, 3] (expected [1, 3]) OK Pre-fix the parent was rewritten to group_name='0:split:[0, 1]' timeout=0:03:42 ranks=[0, 1], "child1.options IS parent.options: True", and all four ranks printed CORRUPT with garbage rank lists. bug4_python_options.py (2 ranks, "cpu:gloo,cuda:nccl") emits no warning and the gloo child keeps the caller's 123 s timeout and group name: CUDA_VISIBLE_DEVICES=2,3 torchrun --nproc-per-node 2 \ triage/bug4_python_options.py child gloo opts type=_Options timeout=0:02:03 name='4a2ad14e...' child nccl opts type=Options timeout=0:02:03 name='4a2ad14e...' both with ranks=[0, 1] child gloo options IS parent gloo options: False parent gloo timeout after split: 0:01:51 ranks=[] Pre-fix this printed the "Tried to pass options to ProcessGroupGloo::split" warning and gave the gloo child 0:30:00 and ''. bug4_gloo_world_deepcopy.py splits instead of raising TypeError: CUDA_VISIBLE_DEVICES=2,3 torchrun --nproc-per-node 2 \ triage/bug4_gloo_world_deepcopy.py OK: child gloo timeout=0:30:00 name='4a2ad14e...' ranks=[0, 1] The 0:30:00 there is not a regression: that script passes no timeout= to split_group, and stock split_group resolves timeout=None to _get_default_timeout(pg_backend) = 30 min before the C++ ever sees it. bug3b_double_split_same_backend.py completes 5/5 with a bare "gloo" world, where it previously hung with rank 0 never returning from split 0: CUDA_VISIBLE_DEVICES=2,3 timeout 180 python \ triage/bug3b_double_split_same_backend.py gloo 5 [rank0] split 0 done ... [rank0] split 4 done [rank1] split 0 done ... [rank1] split 4 done EXIT=0 New tests, all of which fail pre-fix: CUDA_VISIBLE_DEVICES=2,3 python test/distributed/test_c10d_gloo.py \ CommTest.test_split_group_does_not_alias_parent_options \ CommTest.test_split_group_keeps_gloo_options -v Ran 2 tests in 7.291s OK python test/distributed/test_c10d_common.py SplitGroupOptionsTest -v test_split_group_clones_parent_options ... ok test_split_group_opts_apply_to_default_backend_only ... ok Ran 2 tests in 0.900s OK Against the pre-fix binary the same three fail as "unexpectedly identical: ProcessGroupGloo._Options object", "unexpectedly identical: Backend.Options object" and "'caller-supplied' != 'cpu-backend'". Full suites, serially on otherwise-idle GPUs, all green: test_c10d_common.py Ran 72 tests in 113.349s OK test_c10d_gloo.py Ran 289 tests in 1098.597s OK (skipped=4) test_c10d_nccl.py Ran 284 tests in 1645.782s OK (skipped=15) test_c10d_nccl2.py Ran 20 tests in 103.397s OK test_c10d_pybackend.py Ran 5 tests in 4.845s OK test_device_mesh.py Ran 76 tests in 297.845s OK (skipped=5) Three known gaps. mergeRemoteGroup's clone and merge-once path has no multi-device test in tree -- test_c10d_pybackend.py covers the single-device merge only -- so the dedupe branch there is argued from symmetry with split, not measured. The typeid slicing guard is dead code in an in-tree build, since every in-tree Options subclass now overrides clone(); its real consumer is the out-of-tree XPU Options, which cannot be built here, so someone with an XPU build should confirm it compiles and does not silently fall back to aliasing. And stock ProcessGroupNCCL::Options::clone() copies ncclConfig_t shallowly, so a caller-set config.net_name/comm_name (strdup'd in init.cpp) stays shared between parent and child -- unchanged from the __deepcopy__ pybind it replaces, so not a regression; nccl2's clone deep-copies via cloneNcclConfig(), and fixing the stock one belongs to whoever owns ProcessGroupNCCL.cpp. The netName ownership half is not meaningfully testable from Python. The leak is one small allocation on a path reachable only when a caller sets net_name; observing it needs heap accounting (ASAN/valgrind) rather than an assertion, and copy.deepcopy(opts).config.net_name reads the same string before and after. Its real regression test is the existing ProcessGroupNCCL2Test::test_options_type, which is what caught the hand-written-copy-ctor version described above; every split exercises the clone() path itself, with netName == nullptr, and neither crashes nor double-frees. Re-run at the tip of the stack, on 4 idle H100s, with the whole of it applied: CUDA_VISIBLE_DEVICES=0,1,2,3 python test/distributed/test_c10d_nccl2.py -v Ran 24 tests in 135.623s OK CUDA_VISIBLE_DEVICES=0,1,2,3 python test/distributed/test_c10d_nccl.py -v Ran 285 tests in 1797.054s OK (skipped=15) Authored with the assistance of an AI coding agent. Pull Request resolved: pytorch#192110 Approved by: https://github.com/d4l3k ghstack dependencies: pytorch#192104, pytorch#192105, pytorch#192106, pytorch#192107, pytorch#192108, pytorch#192109

`at::divup(x, y)` computes `(x + y - 1) / y`. The addition overflows for large `x` or `y`, and the result then silently rounds **down** instead of up — `divup(2, INT64_MAX)` returns `0`, not `1`. Guarded with `c10::add_overflows`, falling back to a form that cannot overflow. Every caller today passes loop extents and chunk counts, so for most of them this is latent. One caller does reach it: `PoolingParams1D::valid_output_end` bounds the `max_pool1d` output range with `divup`, and a large stride there yields a window starting past the end of the input — a segfault. That is the stacked PR on top of this one; splitting it out because `divup` is used well beyond pooling (softmax, moments, the parallel-loop chunking itself) and deserves its own CI signal. ## Performance Free on the happy path. Measured on an M-series arm64 laptop, `-O3`. Both variants live in **one binary** and are **raced round-robin within each trial**, so thermal drift and frequency changes hit them equally — A/B-ing two libtorch builds measures code layout instead of the change, and gave me a spurious ~9% swing that reproduces as same-build drift. Cycles are from a calibrated dependent-add chain (~4.4 GHz). Numbers below reproduce to three decimals across runs; discard the first run of any such benchmark, it is thermal warm-up. | | old `(x+y-1)/y` | guarded | delta | |---|---|---|---| | **y runtime** (holds an `sdiv`), throughput | 0.448 ns | 0.449 ns | +0.2% | | **y runtime**, latency | 2.629 ns | 2.407 ns | **−8.4%** | | **y compile-time constant** (=16), throughput | 0.171 ns | 0.209 ns | +22.2% | | **y compile-time constant** (=16), latency | 1.330 ns | 1.334 ns | +0.3% | **Runtime divisor — free, and the loop is divider-bound, not issue-bound.** A bare `sdiv` with nothing else in the loop measures **0.449 ns / ~1.9 cyc**, which is exactly what both variants cost, so the guard's extra instructions are absorbed entirely by the divider's throughput. Latency actually *improves*: the guard's `adds` replaces the old `add`/`sub` pair ahead of the divide. ```asm old guarded add x8, x0, x1 sub x8, x1, #1 sub x8, x8, #1 adds x8, x0, x8 sdiv x0, x8, x1 b.vs <fallback> ; predicted not-taken sdiv x0, x8, x1 ``` **Constant divisor — where the cost is.** With `y` a compile-time constant there is no division at all; it folds to a shift, leaving 5 cheap ALU ops, and the guard's 2 extra instructions are a real +22% — of **+0.038 ns/call**. Only `moments_utils.h` (`kChunkSize = 16`) has such a constant among the callers; `SoftMaxKernel.cpp`'s `CHUNK_SIZE` is a runtime `std::min(...)` and takes the free path above. In `moments_utils.h`, `divup` runs once per row of a reduction, ahead of that row's entire Welford loop — in a replica of that shape the difference does not clear the noise floor at any row size. I also tried `x / y + (x % y > 0)`, which needs no guard at all. It is equivalent on throughput but ~34% worse on runtime-divisor latency, because it puts `msub`/`cmp`/`cinc` *after* the divide instead of ahead of it. Happy to switch if the extra header include is a concern. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Pull Request resolved: pytorch#195035 Approved by: https://github.com/Skylion007 Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

zhangxiaoli73 reviewed Sep 9, 2024

View reviewed changes

Comment thread torch/csrc/distributed/c10d/ProcessGroupXCCL.cpp Outdated

zhangxiaoli73 reviewed Sep 9, 2024

View reviewed changes

Comment thread torch/csrc/distributed/c10d/ProcessGroupXCCL.cpp Outdated

zhangxiaoli73 reviewed Sep 9, 2024

View reviewed changes

Comment thread torch/csrc/distributed/c10d/Ops.cpp Outdated

Chao1Han commented Sep 9, 2024

View reviewed changes

Comment thread caffe2/CMakeLists.txt Outdated

guangyey reviewed Sep 9, 2024

View reviewed changes

Comment thread build_variables.bzl Outdated

guangyey reviewed Sep 9, 2024

View reviewed changes

Comment thread cmake/External/xccl.cmake Outdated

gujinghui reviewed Sep 9, 2024

View reviewed changes

Comment thread caffe2/CMakeLists.txt

gujinghui reviewed Sep 9, 2024

View reviewed changes

Comment thread caffe2/CMakeLists.txt Outdated

gujinghui reviewed Sep 9, 2024

View reviewed changes

Comment thread torch/CMakeLists.txt Outdated

zhangxiaoli73 reviewed Sep 10, 2024

View reviewed changes

Comment thread torch/csrc/distributed/c10d/ProcessGroupXCCL.hpp

gujinghui reviewed Sep 11, 2024

View reviewed changes

Comment thread torch/CMakeLists.txt Outdated

gujinghui reviewed Sep 11, 2024

View reviewed changes

Comment thread torch/CMakeLists.txt Outdated

zhangxiaoli73 reviewed Sep 11, 2024

View reviewed changes

Comment thread torch/CMakeLists.txt

zhangxiaoli73 reviewed Sep 11, 2024

View reviewed changes

Comment thread torch/csrc/distributed/c10d/ProcessGroupXCCL.cpp Outdated

zhangxiaoli73 reviewed Sep 11, 2024

View reviewed changes

Comment thread torch/csrc/distributed/c10d/ProcessGroupXCCL.cpp

zhangxiaoli73 reviewed Sep 13, 2024

View reviewed changes

Comment thread torch/csrc/distributed/c10d/ProcessGroupXCCL.cpp Outdated

zhangxiaoli73 reviewed Sep 13, 2024

View reviewed changes

Comment thread torch/csrc/distributed/c10d/ProcessGroupXCCL.cpp Outdated

zhangxiaoli73 reviewed Sep 13, 2024

View reviewed changes

Comment thread torch/csrc/distributed/c10d/ProcessGroupXCCL.cpp

Chao1Han changed the title ~~Xccl~~ Xccl process group for Pytorch Sep 18, 2024

zhangxiaoli73 reviewed Sep 19, 2024

View reviewed changes

bdhirsh and others added 4 commits September 20, 2024 02:54

[CI][CUSPARSELT] Extend cusparselt installation script to support cud…

0e9e7d3

…a 12.6 (pytorch#136321) To prepare for future cuda updates. Pull Request resolved: pytorch#136321 Approved by: https://github.com/Skylion007, https://github.com/eqy

Chao1Han force-pushed the xccl branch from a632efa to c9d6bb8 Compare September 20, 2024 02:55

Chao1Han temporarily deployed to upload-stats September 20, 2024 02:55 — with GitHub Actions Inactive

Chao1Han closed this Sep 23, 2024

Conversation

Chao1Han commented Sep 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zhangxiaoli73 commented Sep 3, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhangxiaoli73 commented Sep 9, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Chao1Han commented Sep 9, 2024

Uh oh!

zhangxiaoli73 Sep 10, 2024

Choose a reason for hiding this comment

Uh oh!

Chao1Han Sep 10, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gujinghui commented Sep 11, 2024

Uh oh!

Chao1Han commented Sep 11, 2024

Uh oh!

zhangxiaoli73 Sep 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhangxiaoli73 commented Sep 11, 2024

Uh oh!

zhangxiaoli73 commented Sep 11, 2024

Uh oh!

Uh oh!

zhangxiaoli73 Sep 13, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zhangxiaoli73 Sep 19, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Chao1Han commented Sep 3, 2024 •

edited

Loading

zhangxiaoli73 Sep 11, 2024 •

edited

Loading