[rank34]:[W1114 14:17:58.414592305 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 34] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank20]:[W1114 14:20:44.179197604 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 20] using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank51]:[W1114 14:20:49.176447164 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 51] using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank26]:[W1114 14:21:01.145147320 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 26] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank39]:[W1114 14:22:23.545425676 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 39] using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank36]:[W1114 14:22:41.190614583 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 36] using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank22]:[W1114 14:24:52.066472973 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 22] using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank32]:[W1114 14:24:56.859690005 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 32] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank2]:[W1114 14:24:58.608611803 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 2] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank56]:[W1114 14:25:18.514248289 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 56] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank47]:[W1114 14:25:26.670207398 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 47] using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank54]:[W1114 14:25:59.264508795 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 54] using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank50]:[W1114 14:26:04.008461129 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 50] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank52]:[W1114 14:26:37.419569947 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 52] using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank48]:[W1114 14:27:17.536760068 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 48] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank49]:[W1114 14:27:26.617245150 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 49] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank34]:[W1114 14:27:59.512113115 socket.cpp:461] [c10d] waitForInput: poll for socket SocketImpl(fd=3, addr=[h100-ord01-03-489]:53314, remote=[h100-ord01-03-801]:29500) returned 0, likely a timeout [rank34]:[W1114 14:27:59.512857061 socket.cpp:486] [c10d] waitForInput: socket SocketImpl(fd=3, addr=[h100-ord01-03-489]:53314, remote=[h100-ord01-03-801]:29500) timed out after 600000ms [rank34]: Traceback (most recent call last): [rank34]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank34]: main() [rank34]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank34]: dist.barrier() [rank34]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank34]: return func(*args, **kwargs) [rank34]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank34]: work = group.barrier(opts=opts) [rank34]: torch.distributed.DistBackendError: [34] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: wait timeout after 600000ms, keys: /default_pg/0//cuda//0 [rank34]: Exception raised from doWait at ../torch/csrc/distributed/c10d/TCPStore.cpp:587 (most recent call first): [rank34]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f014816bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank34]: frame #1: + 0x15eeaa9 (0x7f01332aeaa9 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #2: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f0137cdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #3: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f0137cdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #4: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0137c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0137c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f0137c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #7: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f00ea95adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank34]: frame #8: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f00ea9674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank34]: frame #9: + 0x1205255 (0x7f00ea96d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank34]: frame #10: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f00ea96e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank34]: frame #11: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f00ea97f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank34]: frame #12: + 0x5fc28a2 (0x7f0137c828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #13: + 0x5fcd035 (0x7f0137c8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #14: + 0x55d8ceb (0x7f0137298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #15: + 0x55d6579 (0x7f0137296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #16: + 0x1aa7828 (0x7f0133767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #17: + 0x5fd6d74 (0x7f0137c96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #18: + 0x5fd7b05 (0x7f0137c97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank34]: frame #19: + 0xe1df28 (0x7f0147783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank34]: frame #20: + 0x4cf144 (0x7f0146e35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank34]: frame #21: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank34]: frame #22: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #23: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank34]: frame #24: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #25: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #26: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #27: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #28: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #29: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #30: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #31: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank34]: frame #32: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #33: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank34]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank34]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank34]: frame #36: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #37: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #38: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #39: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank34]: frame #40: + 0x29d90 (0x7f015e18ed90 in /lib/x86_64-linux-gnu/libc.so.6) [rank34]: frame #41: __libc_start_main + 0x80 (0x7f015e18ee40 in /lib/x86_64-linux-gnu/libc.so.6) [rank34]: frame #42: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank34]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank5]:[W1114 14:28:02.674686463 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 5] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. srun: error: h100-ord01-03-489: task 34: Exited with exit code 1 [rank58]:[W1114 14:28:47.706580434 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 58] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank6]:[W1114 14:29:02.552959259 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 6] using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank12]:[W1114 14:29:03.319697460 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 12] using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank4]:[W1114 14:29:50.763990416 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 4] using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank38]:[W1114 14:30:00.799576323 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 38] using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank0]:[W1114 14:30:02.196696316 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank0]: Traceback (most recent call last): [rank0]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank0]: main() [rank0]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank0]: dist.barrier() [rank0]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank0]: return func(*args, **kwargs) [rank0]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank0]: work = group.barrier(opts=opts) [rank0]: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:2329, internal error - please report this issue to the NCCL developers, NCCL version 2.21.5 [rank0]: ncclInternalError: Internal check failed. [rank0]: Last error: [rank0]: Bootstrap : no socket interface found [rank0]:[W1114 14:30:03.882684811 ProcessGroupNCCL.cpp:1326] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) [rank2]: Traceback (most recent call last): [rank2]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank2]: main() [rank2]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank2]: dist.barrier() [rank2]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank2]: return func(*args, **kwargs) [rank2]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank2]: work = group.barrier(opts=opts) [rank2]: torch.distributed.DistBackendError: [2] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank2]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank2]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f2edef6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank2]: frame #1: + 0x6021e58 (0x7f2eceae1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f2eceadebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f2eceadef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f2eceadfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2ecea8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2ecea8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2ecea8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f2e8175adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank2]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f2e817674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank2]: frame #10: + 0x1205255 (0x7f2e8176d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank2]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f2e8176e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank2]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f2e8177f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank2]: frame #13: + 0x5fc28a2 (0x7f2ecea828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #14: + 0x5fcd035 (0x7f2ecea8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #15: + 0x55d8ceb (0x7f2ece098ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #16: + 0x55d6579 (0x7f2ece096579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #17: + 0x1aa7828 (0x7f2eca567828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #18: + 0x5fd6d74 (0x7f2ecea96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #19: + 0x5fd7b05 (0x7f2ecea97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank2]: frame #20: + 0xe1df28 (0x7f2ede583f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank2]: frame #21: + 0x4cf144 (0x7f2eddc35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank2]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank2]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank2]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank2]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank2]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank2]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank2]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank2]: frame #41: + 0x29d90 (0x7f2ef4f81d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank2]: frame #42: __libc_start_main + 0x80 (0x7f2ef4f81e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank2]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank2]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank6]: Traceback (most recent call last): [rank6]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank6]: main() [rank6]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank6]: dist.barrier() [rank6]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank6]: return func(*args, **kwargs) [rank6]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank6]: work = group.barrier(opts=opts) [rank6]: torch.distributed.DistBackendError: [6] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank6]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank6]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fe290cb8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank6]: frame #1: + 0x6021e58 (0x7fe2808e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fe2808debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fe2808def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fe2808dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe28088ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe28088ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fe28088ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fe23355adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank6]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fe2335674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank6]: frame #10: + 0x1205255 (0x7fe23356d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank6]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fe23356e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank6]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fe23357f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank6]: frame #13: + 0x5fc28a2 (0x7fe2808828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #14: + 0x5fcd035 (0x7fe28088d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #15: + 0x55d8ceb (0x7fe27fe98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: Traceback (most recent call last): [rank5]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank5]: main() [rank5]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank5]: dist.barrier() [rank5]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank5]: return func(*args, **kwargs) [rank5]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank5]: work = group.barrier(opts=opts) [rank5]: torch.distributed.DistBackendError: [5] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank5]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank5]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fccb60b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank5]: frame #1: + 0x6021e58 (0x7fcca5ce1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fcca5cdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fcca5cdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fcca5cdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fcca5c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fcca5c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fcca5c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fcc5895adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank5]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fcc589674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank5]: frame #10: + 0x1205255 (0x7fcc5896d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank5]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fcc5896e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank5]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fcc5897f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank5]: frame #13: + 0x5fc28a2 (0x7fcca5c828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #14: + 0x5fcd035 (0x7fcca5c8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #15: + 0x55d8ceb (0x7fcca5298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #16: + 0x55d6579 (0x7fe27fe96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #17: + 0x1aa7828 (0x7fe27c367828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #18: + 0x5fd6d74 (0x7fe280896d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #19: + 0x5fd7b05 (0x7fe280897b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank6]: frame #20: + 0xe1df28 (0x7fe290383f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank6]: frame #21: + 0x4cf144 (0x7fe28fa35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank6]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank6]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank6]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank6]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank6]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank6]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank6]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank6]: frame #41: + 0x29d90 (0x7fe2a6c96d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank6]: frame #42: __libc_start_main + 0x80 (0x7fe2a6c96e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank6]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank6]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank4]: Traceback (most recent call last): [rank4]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank4]: main() [rank4]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank4]: dist.barrier() [rank4]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank4]: return func(*args, **kwargs) [rank4]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank4]: work = group.barrier(opts=opts) [rank4]: torch.distributed.DistBackendError: [4] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank4]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank4]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f9b6b6d3cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank4]: frame #1: + 0x6021e58 (0x7f9bb9ce1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f9bb9cdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f9bb9cdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f9bb9cdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9bb9c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9bb9c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9bb9c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f9b6c95adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank4]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f9b6c9674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank4]: frame #10: + 0x1205255 (0x7f9b6c96d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank4]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f9b6c96e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank4]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f9b6c97f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank4]: frame #13: + 0x5fc28a2 (0x7f9bb9c828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #14: + 0x5fcd035 (0x7f9bb9c8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #15: + 0x55d8ceb (0x7f9bb9298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #16: + 0x55d6579 (0x7fcca5296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #17: + 0x1aa7828 (0x7fcca1767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #18: + 0x5fd6d74 (0x7fcca5c96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #19: + 0x5fd7b05 (0x7fcca5c97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank5]: frame #20: + 0xe1df28 (0x7fccb5783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank5]: frame #21: + 0x4cf144 (0x7fccb4e35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank5]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank5]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank5]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank5]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank5]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank5]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank5]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank5]: frame #41: + 0x29d90 (0x7fcccc106d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank5]: frame #42: __libc_start_main + 0x80 (0x7fcccc106e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank5]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank5]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank4]: frame #16: + 0x55d6579 (0x7f9bb9296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #17: + 0x1aa7828 (0x7f9bb5767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #18: + 0x5fd6d74 (0x7f9bb9c96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #19: + 0x5fd7b05 (0x7f9bb9c97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank4]: frame #20: + 0xe1df28 (0x7f9bc9783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank4]: frame #21: + 0x4cf144 (0x7f9bc8e35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank4]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank4]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank4]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank4]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank4]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank4]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank4]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank4]: frame #41: + 0x29d90 (0x7f9be003fd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank4]: frame #42: __libc_start_main + 0x80 (0x7f9be003fe40 in /lib/x86_64-linux-gnu/libc.so.6) [rank4]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank4]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank32]: Traceback (most recent call last): [rank32]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank32]: main() [rank32]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank32]: dist.barrier() [rank32]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank32]: return func(*args, **kwargs) [rank32]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank32]: work = group.barrier(opts=opts) [rank32]: torch.distributed.DistBackendError: [32] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank32]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank32]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f98cccb8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank32]: frame #1: + 0x6021e58 (0x7f98bc8e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f98bc8debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f98bc8def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f98bc8dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f98bc88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f98bc88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f98bc88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f986f55adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank32]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f986f5674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank32]: frame #10: + 0x1205255 (0x7f986f56d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank32]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f986f56e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank32]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f986f57f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank32]: frame #13: + 0x5fc28a2 (0x7f98bc8828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #14: + 0x5fcd035 (0x7f98bc88d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #15: + 0x55d8ceb (0x7f98bbe98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: Traceback (most recent call last): [rank36]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank36]: main() [rank36]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank36]: dist.barrier() [rank36]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank36]: return func(*args, **kwargs) [rank36]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank36]: work = group.barrier(opts=opts) [rank36]: torch.distributed.DistBackendError: [36] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank36]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank36]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f1d8cb6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank36]: frame #1: + 0x6021e58 (0x7f1d7c6e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f1d7c6debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f1d7c6def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f1d7c6dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1d7c68ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1d7c68ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1d7c68ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f1d2f35adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank36]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f1d2f3674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank36]: frame #10: + 0x1205255 (0x7f1d2f36d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank36]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f1d2f36e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank36]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f1d2f37f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank36]: frame #13: + 0x5fc28a2 (0x7f1d7c6828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #14: + 0x5fcd035 (0x7f1d7c68d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #15: + 0x55d8ceb (0x7f1d7bc98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: Traceback (most recent call last): [rank39]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank39]: main() [rank39]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank39]: dist.barrier() [rank39]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank39]: return func(*args, **kwargs) [rank39]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank39]: work = group.barrier(opts=opts) [rank39]: torch.distributed.DistBackendError: [39] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank39]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank39]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f7c550b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank39]: frame #1: + 0x6021e58 (0x7f7c44ce1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f7c44cdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f7c44cdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f7c44cdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7c44c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7c44c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7c44c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f7bf795adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank39]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f7bf79674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank39]: frame #10: + 0x1205255 (0x7f7bf796d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank39]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f7bf796e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank39]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f7bf797f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank39]: frame #13: + 0x5fc28a2 (0x7f7c44c828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #14: + 0x5fcd035 (0x7f7c44c8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #15: + 0x55d8ceb (0x7f7c44298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #16: + 0x55d6579 (0x7f98bbe96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #17: + 0x1aa7828 (0x7f98b8367828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #18: + 0x5fd6d74 (0x7f98bc896d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #19: + 0x5fd7b05 (0x7f98bc897b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank32]: frame #20: + 0xe1df28 (0x7f98cc383f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank32]: frame #21: + 0x4cf144 (0x7f98cba35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank32]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank32]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank32]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank32]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank32]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank32]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank32]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank32]: frame #41: + 0x29d90 (0x7f98e2ccad90 in /lib/x86_64-linux-gnu/libc.so.6) [rank32]: frame #42: __libc_start_main + 0x80 (0x7f98e2ccae40 in /lib/x86_64-linux-gnu/libc.so.6) [rank32]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank32]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank36]: frame #16: + 0x55d6579 (0x7f1d7bc96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #17: + 0x1aa7828 (0x7f1d78167828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #18: + 0x5fd6d74 (0x7f1d7c696d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #19: + 0x5fd7b05 (0x7f1d7c697b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank36]: frame #20: + 0xe1df28 (0x7f1d8c183f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank36]: frame #21: + 0x4cf144 (0x7f1d8b835144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank36]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank36]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank36]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank36]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank36]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank36]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank36]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank36]: frame #41: + 0x29d90 (0x7f1da2b77d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank36]: frame #42: __libc_start_main + 0x80 (0x7f1da2b77e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank36]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank36]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank38]: Traceback (most recent call last): [rank38]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank38]: main() [rank38]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank38]: dist.barrier() [rank38]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank38]: return func(*args, **kwargs) [rank38]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank38]: work = group.barrier(opts=opts) [rank38]: torch.distributed.DistBackendError: [38] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank38]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank38]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f44274b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank38]: frame #1: + 0x6021e58 (0x7f44170e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f44170debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f44170def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f44170dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f441708ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f441708ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f441708ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f43c9d5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank38]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f43c9d674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank38]: frame #10: + 0x1205255 (0x7f43c9d6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank38]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f43c9d6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank38]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f43c9d7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank38]: frame #13: + 0x5fc28a2 (0x7f44170828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #14: + 0x5fcd035 (0x7f441708d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #15: + 0x55d8ceb (0x7f4416698ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #16: + 0x55d6579 (0x7f7c44296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #17: + 0x1aa7828 (0x7f7c40767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #18: + 0x5fd6d74 (0x7f7c44c96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #19: + 0x5fd7b05 (0x7f7c44c97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank39]: frame #20: + 0xe1df28 (0x7f7c54783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank39]: frame #21: + 0x4cf144 (0x7f7c53e35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank39]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank39]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank39]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank39]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank39]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank39]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank39]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank39]: frame #41: + 0x29d90 (0x7f7c6b088d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank39]: frame #42: __libc_start_main + 0x80 (0x7f7c6b088e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank39]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank39]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank38]: frame #16: + 0x55d6579 (0x7f4416696579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #17: + 0x1aa7828 (0x7f4412b67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #18: + 0x5fd6d74 (0x7f4417096d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #19: + 0x5fd7b05 (0x7f4417097b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank38]: frame #20: + 0xe1df28 (0x7f4426b83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank38]: frame #21: + 0x4cf144 (0x7f4426235144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank38]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank38]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank38]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank38]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank38]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank38]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank38]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank38]: frame #41: + 0x29d90 (0x7f443d4b8d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank38]: frame #42: __libc_start_main + 0x80 (0x7f443d4b8e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank38]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank38]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank47]: Traceback (most recent call last): [rank47]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank47]: main() [rank47]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank47]: dist.barrier() [rank47]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank47]: return func(*args, **kwargs) [rank47]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank47]: work = group.barrier(opts=opts) [rank47]: torch.distributed.DistBackendError: [47] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank47]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank47]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f2fb196bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank47]: frame #1: + 0x6021e58 (0x7f2fa14e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f2fa14debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f2fa14def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f2fa14dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2fa148ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2fa148ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2fa148ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f2f5415adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank47]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f2f541674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank47]: frame #10: + 0x1205255 (0x7f2f5416d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank47]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f2f5416e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank47]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f2f5417f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank47]: frame #13: + 0x5fc28a2 (0x7f2fa14828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #14: + 0x5fcd035 (0x7f2fa148d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #15: + 0x55d8ceb (0x7f2fa0a98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #16: + 0x55d6579 (0x7f2fa0a96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #17: + 0x1aa7828 (0x7f2f9cf67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #18: + 0x5fd6d74 (0x7f2fa1496d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #19: + 0x5fd7b05 (0x7f2fa1497b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank47]: frame #20: + 0xe1df28 (0x7f2fb0f83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank47]: frame #21: + 0x4cf144 (0x7f2fb0635144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank47]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank47]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank47]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank47]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank47]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank47]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank47]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank47]: frame #41: + 0x29d90 (0x7f2fc79e0d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank47]: frame #42: __libc_start_main + 0x80 (0x7f2fc79e0e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank47]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank47]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank56]: Traceback (most recent call last): [rank56]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank56]: main() [rank56]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank56]: dist.barrier() [rank56]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank56]: return func(*args, **kwargs) [rank56]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank56]: work = group.barrier(opts=opts) [rank56]: torch.distributed.DistBackendError: [56] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank56]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank56]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f7bdc2b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank56]: frame #1: + 0x6021e58 (0x7f7bcbee1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f7bcbedebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f7bcbedef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f7bcbedfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7bcbe8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7bcbe8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7bcbe8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f7b7eb5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank56]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f7b7eb674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank56]: frame #10: + 0x1205255 (0x7f7b7eb6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank56]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f7b7eb6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank56]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f7b7eb7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank56]: frame #13: + 0x5fc28a2 (0x7f7bcbe828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #14: + 0x5fcd035 (0x7f7bcbe8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #15: + 0x55d8ceb (0x7f7bcb498ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: Traceback (most recent call last): [rank58]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank58]: main() [rank58]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank58]: dist.barrier() [rank58]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank58]: return func(*args, **kwargs) [rank58]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank58]: work = group.barrier(opts=opts) [rank58]: torch.distributed.DistBackendError: [58] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank58]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank58]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f7f4ceb8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank58]: frame #1: + 0x6021e58 (0x7f7f3cae1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f7f3cadebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f7f3cadef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f7f3cadfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7f3ca8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7f3ca8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7f3ca8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f7eef75adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank58]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f7eef7674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank58]: frame #10: + 0x1205255 (0x7f7eef76d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank58]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f7eef76e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank58]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f7eef77f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank58]: frame #13: + 0x5fc28a2 (0x7f7f3ca828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #14: + 0x5fcd035 (0x7f7f3ca8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #15: + 0x55d8ceb (0x7f7f3c098ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #16: + 0x55d6579 (0x7f7bcb496579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #17: + 0x1aa7828 (0x7f7bc7967828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #18: + 0x5fd6d74 (0x7f7bcbe96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #19: + 0x5fd7b05 (0x7f7bcbe97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank56]: frame #20: + 0xe1df28 (0x7f7bdb983f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank56]: frame #21: + 0x4cf144 (0x7f7bdb035144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank56]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank56]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank56]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank56]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank56]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank56]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank56]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank56]: frame #41: + 0x29d90 (0x7f7bf22fdd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank56]: frame #42: __libc_start_main + 0x80 (0x7f7bf22fde40 in /lib/x86_64-linux-gnu/libc.so.6) [rank56]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank56]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank58]: frame #16: + 0x55d6579 (0x7f7f3c096579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #17: + 0x1aa7828 (0x7f7f38567828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #18: + 0x5fd6d74 (0x7f7f3ca96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #19: + 0x5fd7b05 (0x7f7f3ca97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank58]: frame #20: + 0xe1df28 (0x7f7f4c583f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank58]: frame #21: + 0x4cf144 (0x7f7f4bc35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank58]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank58]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank58]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank58]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank58]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank58]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank58]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank58]: frame #41: + 0x29d90 (0x7f7f62e83d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank58]: frame #42: __libc_start_main + 0x80 (0x7f7f62e83e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank58]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank58]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank20]: Traceback (most recent call last): [rank20]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank20]: main() [rank20]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank20]: dist.barrier() [rank20]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank20]: return func(*args, **kwargs) [rank20]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank20]: work = group.barrier(opts=opts) [rank20]: torch.distributed.DistBackendError: [20] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank20]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank20]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f94e0b6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank20]: frame #1: + 0x6021e58 (0x7f94d06e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f94d06debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f94d06def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f94d06dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f94d068ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f94d068ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f94d068ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f948335adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank20]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f94833674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank20]: frame #10: + 0x1205255 (0x7f948336d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank20]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f948336e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank20]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f948337f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank20]: frame #13: + 0x5fc28a2 (0x7f94d06828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #14: + 0x5fcd035 (0x7f94d068d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #15: + 0x55d8ceb (0x7f94cfc98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: Traceback (most recent call last): [rank22]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank22]: main() [rank22]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank22]: dist.barrier() [rank22]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank22]: return func(*args, **kwargs) [rank22]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank22]: work = group.barrier(opts=opts) [rank22]: torch.distributed.DistBackendError: [22] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank22]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank22]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd5a3ad3cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank22]: frame #1: + 0x6021e58 (0x7fd5f20e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fd5f20debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fd5f20def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fd5f20dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd5f208ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd5f208ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd5f208ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fd5a4d5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank22]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fd5a4d674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank22]: frame #10: + 0x1205255 (0x7fd5a4d6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank22]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fd5a4d6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank22]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fd5a4d7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank22]: frame #13: + 0x5fc28a2 (0x7fd5f20828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #14: + 0x5fcd035 (0x7fd5f208d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #15: + 0x55d8ceb (0x7fd5f1698ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #16: + 0x55d6579 (0x7f94cfc96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #17: + 0x1aa7828 (0x7f94cc167828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #18: + 0x5fd6d74 (0x7f94d0696d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #19: + 0x5fd7b05 (0x7f94d0697b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank20]: frame #20: + 0xe1df28 (0x7f94e0183f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank20]: frame #21: + 0x4cf144 (0x7f94df835144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank20]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank20]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank20]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank20]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank20]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank20]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank20]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank20]: frame #41: + 0x29d90 (0x7f94f6bf7d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank20]: frame #42: __libc_start_main + 0x80 (0x7f94f6bf7e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank20]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank20]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank22]: frame #16: + 0x55d6579 (0x7fd5f1696579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #17: + 0x1aa7828 (0x7fd5edb67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #18: + 0x5fd6d74 (0x7fd5f2096d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #19: + 0x5fd7b05 (0x7fd5f2097b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank22]: frame #20: + 0xe1df28 (0x7fd601b83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank22]: frame #21: + 0x4cf144 (0x7fd601235144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank22]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank22]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank22]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank22]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank22]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank22]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank22]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank22]: frame #41: + 0x29d90 (0x7fd61842fd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank22]: frame #42: __libc_start_main + 0x80 (0x7fd61842fe40 in /lib/x86_64-linux-gnu/libc.so.6) [rank22]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank22]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank26]: Traceback (most recent call last): [rank26]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank26]: main() [rank26]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank26]: dist.barrier() [rank26]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank26]: return func(*args, **kwargs) [rank26]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank26]: work = group.barrier(opts=opts) [rank26]: torch.distributed.DistBackendError: [26] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank26]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank26]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7feda936bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank26]: frame #1: + 0x6021e58 (0x7fed98ee1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fed98edebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fed98edef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fed98edfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fed98e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fed98e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fed98e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fed4bb5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank26]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fed4bb674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank26]: frame #10: + 0x1205255 (0x7fed4bb6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank26]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fed4bb6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank26]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fed4bb7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank26]: frame #13: + 0x5fc28a2 (0x7fed98e828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #14: + 0x5fcd035 (0x7fed98e8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #15: + 0x55d8ceb (0x7fed98498ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #16: + 0x55d6579 (0x7fed98496579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #17: + 0x1aa7828 (0x7fed94967828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #18: + 0x5fd6d74 (0x7fed98e96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #19: + 0x5fd7b05 (0x7fed98e97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank26]: frame #20: + 0xe1df28 (0x7feda8983f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank26]: frame #21: + 0x4cf144 (0x7feda8035144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank26]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank26]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank26]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank26]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank26]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank26]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank26]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank26]: frame #41: + 0x29d90 (0x7fedbf351d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank26]: frame #42: __libc_start_main + 0x80 (0x7fedbf351e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank26]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank26]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank12]: Traceback (most recent call last): [rank12]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank12]: main() [rank12]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank12]: dist.barrier() [rank12]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank12]: return func(*args, **kwargs) [rank12]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank12]: work = group.barrier(opts=opts) [rank12]: torch.distributed.DistBackendError: [12] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank12]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank12]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f744c4d3cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank12]: frame #1: + 0x6021e58 (0x7f749aae1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f749aadebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f749aadef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f749aadfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f749aa8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f749aa8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f749aa8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f744d75adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank12]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f744d7674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank12]: frame #10: + 0x1205255 (0x7f744d76d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank12]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f744d76e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank12]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f744d77f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank12]: frame #13: + 0x5fc28a2 (0x7f749aa828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #14: + 0x5fcd035 (0x7f749aa8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #15: + 0x55d8ceb (0x7f749a098ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #16: + 0x55d6579 (0x7f749a096579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #17: + 0x1aa7828 (0x7f7496567828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #18: + 0x5fd6d74 (0x7f749aa96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #19: + 0x5fd7b05 (0x7f749aa97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank12]: frame #20: + 0xe1df28 (0x7f74aa583f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank12]: frame #21: + 0x4cf144 (0x7f74a9c35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank12]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank12]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank12]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank12]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank12]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank12]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank12]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank12]: frame #41: + 0x29d90 (0x7f74c0e49d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank12]: frame #42: __libc_start_main + 0x80 (0x7f74c0e49e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank12]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank12]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank48]: Traceback (most recent call last): [rank48]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank48]: main() [rank48]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank48]: dist.barrier() [rank48]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank48]: return func(*args, **kwargs) [rank48]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank48]: work = group.barrier(opts=opts) [rank48]: torch.distributed.DistBackendError: [48] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank48]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank48]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f844f2b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank48]: frame #1: + 0x6021e58 (0x7f843eee1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f843eedebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f843eedef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f843eedfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f843ee8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f843ee8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f843ee8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f83f1b5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank48]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f83f1b674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank48]: frame #10: + 0x1205255 (0x7f83f1b6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank48]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f83f1b6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank48]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f83f1b7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank48]: frame #13: + 0x5fc28a2 (0x7f843ee828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #14: + 0x5fcd035 (0x7f843ee8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #15: + 0x55d8ceb (0x7f843e498ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: Traceback (most recent call last): [rank50]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank50]: main() [rank50]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank50]: dist.barrier() [rank50]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank50]: return func(*args, **kwargs) [rank50]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank50]: work = group.barrier(opts=opts) [rank50]: torch.distributed.DistBackendError: [50] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank50]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank50]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f4cf956bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank50]: frame #1: + 0x6021e58 (0x7f4ce90e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f4ce90debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f4ce90def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f4ce90dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4ce908ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4ce908ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f4ce908ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f4c9bd5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank50]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f4c9bd674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank50]: frame #10: + 0x1205255 (0x7f4c9bd6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank50]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f4c9bd6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank50]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f4c9bd7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank50]: frame #13: + 0x5fc28a2 (0x7f4ce90828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #14: + 0x5fcd035 (0x7f4ce908d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #15: + 0x55d8ceb (0x7f4ce8698ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: Traceback (most recent call last): [rank52]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank52]: main() [rank52]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank52]: dist.barrier() [rank52]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank52]: return func(*args, **kwargs) [rank52]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank52]: work = group.barrier(opts=opts) [rank52]: torch.distributed.DistBackendError: [52] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank52]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank52]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd25bf6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank52]: frame #1: + 0x6021e58 (0x7fd24bae1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fd24badebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fd24badef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fd24badfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd24ba8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd24ba8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd24ba8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fd1fe75adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank52]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fd1fe7674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank52]: frame #10: + 0x1205255 (0x7fd1fe76d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank52]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fd1fe76e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank52]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fd1fe77f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank52]: frame #13: + 0x5fc28a2 (0x7fd24ba828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #14: + 0x5fcd035 (0x7fd24ba8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #15: + 0x55d8ceb (0x7fd24b098ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: Traceback (most recent call last): [rank54]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank54]: main() [rank54]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank54]: dist.barrier() [rank54]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank54]: return func(*args, **kwargs) [rank54]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank54]: work = group.barrier(opts=opts) [rank54]: torch.distributed.DistBackendError: [54] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank54]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank54]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f76a00b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank54]: frame #1: + 0x6021e58 (0x7f768fce1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f768fcdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f768fcdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f768fcdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f768fc8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f768fc8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f768fc8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f764295adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank54]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f76429674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank54]: frame #10: + 0x1205255 (0x7f764296d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank54]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f764296e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank54]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f764297f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank54]: frame #13: + 0x5fc28a2 (0x7f768fc828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #14: + 0x5fcd035 (0x7f768fc8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #15: + 0x55d8ceb (0x7f768f298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #16: + 0x55d6579 (0x7f843e496579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #17: + 0x1aa7828 (0x7f843a967828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #18: + 0x5fd6d74 (0x7f843ee96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #19: + 0x5fd7b05 (0x7f843ee97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank48]: frame #20: + 0xe1df28 (0x7f844e983f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank48]: frame #21: + 0x4cf144 (0x7f844e035144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank48]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank48]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank48]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank48]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank48]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank48]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank48]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank48]: frame #41: + 0x29d90 (0x7f84652e9d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank48]: frame #42: __libc_start_main + 0x80 (0x7f84652e9e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank48]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank48]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank49]: Traceback (most recent call last): [rank49]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank49]: main() [rank49]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank49]: dist.barrier() [rank49]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank49]: return func(*args, **kwargs) [rank49]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank49]: work = group.barrier(opts=opts) [rank49]: torch.distributed.DistBackendError: [49] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank49]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank49]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7ff4d836bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank49]: frame #1: + 0x6021e58 (0x7ff4c7ee1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7ff4c7edebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7ff4c7edef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7ff4c7edfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff4c7e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff4c7e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff4c7e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7ff47ab5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank49]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7ff47ab674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank49]: frame #10: + 0x1205255 (0x7ff47ab6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank49]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7ff47ab6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank49]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7ff47ab7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank49]: frame #13: + 0x5fc28a2 (0x7ff4c7e828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #14: + 0x5fcd035 (0x7ff4c7e8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #15: + 0x55d8ceb (0x7ff4c7498ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #16: + 0x55d6579 (0x7f4ce8696579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #17: + 0x1aa7828 (0x7f4ce4b67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #18: + 0x5fd6d74 (0x7f4ce9096d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #19: + 0x5fd7b05 (0x7f4ce9097b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank50]: frame #20: + 0xe1df28 (0x7f4cf8b83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank50]: frame #21: + 0x4cf144 (0x7f4cf8235144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank50]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank50]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank50]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank50]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank50]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank50]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank50]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank50]: frame #41: + 0x29d90 (0x7f4d0f5ced90 in /lib/x86_64-linux-gnu/libc.so.6) [rank50]: frame #42: __libc_start_main + 0x80 (0x7f4d0f5cee40 in /lib/x86_64-linux-gnu/libc.so.6) [rank50]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank50]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank51]: Traceback (most recent call last): [rank51]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank51]: main() [rank51]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank51]: dist.barrier() [rank51]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank51]: return func(*args, **kwargs) [rank51]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank51]: work = group.barrier(opts=opts) [rank51]: torch.distributed.DistBackendError: [51] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank51]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank51]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f9f20ab8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank51]: frame #1: + 0x6021e58 (0x7f9f106e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f9f106debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f9f106def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f9f106dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9f1068ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9f1068ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f9f1068ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f9ec335adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank51]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f9ec33674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank51]: frame #10: + 0x1205255 (0x7f9ec336d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank51]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f9ec336e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank51]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f9ec337f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank51]: frame #13: + 0x5fc28a2 (0x7f9f106828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #14: + 0x5fcd035 (0x7f9f1068d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #15: + 0x55d8ceb (0x7f9f0fc98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #16: + 0x55d6579 (0x7fd24b096579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #17: + 0x1aa7828 (0x7fd247567828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #18: + 0x5fd6d74 (0x7fd24ba96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #19: + 0x5fd7b05 (0x7fd24ba97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank52]: frame #20: + 0xe1df28 (0x7fd25b583f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank52]: frame #21: + 0x4cf144 (0x7fd25ac35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank52]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank52]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank52]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank52]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank52]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank52]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank52]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank52]: frame #41: + 0x29d90 (0x7fd271f67d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank52]: frame #42: __libc_start_main + 0x80 (0x7fd271f67e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank52]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank52]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank54]: frame #16: + 0x55d6579 (0x7f768f296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #17: + 0x1aa7828 (0x7f768b767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #18: + 0x5fd6d74 (0x7f768fc96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #19: + 0x5fd7b05 (0x7f768fc97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank54]: frame #20: + 0xe1df28 (0x7f769f783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank54]: frame #21: + 0x4cf144 (0x7f769ee35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank54]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank54]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank54]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank54]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank54]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank54]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank54]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank54]: frame #41: + 0x29d90 (0x7f76b6076d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank54]: frame #42: __libc_start_main + 0x80 (0x7f76b6076e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank54]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank54]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank49]: frame #16: + 0x55d6579 (0x7ff4c7496579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #17: + 0x1aa7828 (0x7ff4c3967828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #18: + 0x5fd6d74 (0x7ff4c7e96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #19: + 0x5fd7b05 (0x7ff4c7e97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank49]: frame #20: + 0xe1df28 (0x7ff4d7983f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank49]: frame #21: + 0x4cf144 (0x7ff4d7035144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank49]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank49]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank49]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank49]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank49]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank49]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank49]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank49]: frame #41: + 0x29d90 (0x7ff4ee379d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank49]: frame #42: __libc_start_main + 0x80 (0x7ff4ee379e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank49]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank49]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank51]: frame #16: + 0x55d6579 (0x7f9f0fc96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #17: + 0x1aa7828 (0x7f9f0c167828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #18: + 0x5fd6d74 (0x7f9f10696d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #19: + 0x5fd7b05 (0x7f9f10697b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank51]: frame #20: + 0xe1df28 (0x7f9f20183f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank51]: frame #21: + 0x4cf144 (0x7f9f1f835144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank51]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank51]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank51]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank51]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank51]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank51]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank51]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank51]: frame #41: + 0x29d90 (0x7f9f36a63d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank51]: frame #42: __libc_start_main + 0x80 (0x7f9f36a63e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank51]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank51]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank32]:[W1114 14:30:04.565347075 ProcessGroupNCCL.cpp:1326] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) [rank56]:[W1114 14:30:04.498631012 ProcessGroupNCCL.cpp:1326] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) [rank48]:[W1114 14:30:04.070178798 ProcessGroupNCCL.cpp:1326] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) srun: error: h100-ord01-03-489: tasks 38-39: Exited with exit code 1 srun: error: h100-ord01-03-801: tasks 0,6: Exited with exit code 1 srun: error: h100-ord01-03-489: tasks 32,36: Exited with exit code 1 srun: error: h100-ord01-03-801: tasks 2,4: Exited with exit code 1 srun: error: h100-ord01-03-801: task 5: Exited with exit code 1 srun: error: h100-ord01-03-1023: task 26: Exited with exit code 1 srun: error: h100-ord01-03-948: tasks 56,58: Exited with exit code 1 srun: error: h100-ord01-03-1017: task 47: Exited with exit code 1 srun: error: h100-ord01-03-729: tasks 20,22: Exited with exit code 1 srun: error: h100-ord01-03-313: tasks 48,54: Exited with exit code 1 srun: error: h100-ord01-03-629: task 12: Exited with exit code 1 srun: error: h100-ord01-03-313: tasks 49,52: Exited with exit code 1 srun: error: h100-ord01-03-313: tasks 50-51: Exited with exit code 1 [rank28]:[W1114 14:31:04.333466537 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 28] using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank28]: Traceback (most recent call last): [rank28]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank28]: main() [rank28]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank28]: dist.barrier() [rank28]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank28]: return func(*args, **kwargs) [rank28]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank28]: work = group.barrier(opts=opts) [rank28]: torch.distributed.DistBackendError: [28] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank28]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank28]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f6f8736bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank28]: frame #1: + 0x6021e58 (0x7f6f76ee1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f6f76edebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f6f76edef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f6f76edfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6f76e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6f76e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6f76e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f6f29b5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank28]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f6f29b674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank28]: frame #10: + 0x1205255 (0x7f6f29b6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank28]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f6f29b6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank28]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f6f29b7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank28]: frame #13: + 0x5fc28a2 (0x7f6f76e828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #14: + 0x5fcd035 (0x7f6f76e8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #15: + 0x55d8ceb (0x7f6f76498ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #16: + 0x55d6579 (0x7f6f76496579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #17: + 0x1aa7828 (0x7f6f72967828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #18: + 0x5fd6d74 (0x7f6f76e96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #19: + 0x5fd7b05 (0x7f6f76e97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank28]: frame #20: + 0xe1df28 (0x7f6f86983f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank28]: frame #21: + 0x4cf144 (0x7f6f86035144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank28]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank28]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank28]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank28]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank28]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank28]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank28]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank28]: frame #41: + 0x29d90 (0x7f6f9d367d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank28]: frame #42: __libc_start_main + 0x80 (0x7f6f9d367e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank28]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank28]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-1023: task 28: Exited with exit code 1 [rank13]:[W1114 14:31:13.464049868 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 13] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank13]: Traceback (most recent call last): [rank13]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank13]: main() [rank13]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank13]: dist.barrier() [rank13]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank13]: return func(*args, **kwargs) [rank13]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank13]: work = group.barrier(opts=opts) [rank13]: torch.distributed.DistBackendError: [13] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank13]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank13]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd46096bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank13]: frame #1: + 0x6021e58 (0x7fd4504e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fd4504debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fd4504def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fd4504dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd45048ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd45048ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd45048ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fd40315adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank13]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fd4031674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank13]: frame #10: + 0x1205255 (0x7fd40316d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank13]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fd40316e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank13]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fd40317f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank13]: frame #13: + 0x5fc28a2 (0x7fd4504828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #14: + 0x5fcd035 (0x7fd45048d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #15: + 0x55d8ceb (0x7fd44fa98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #16: + 0x55d6579 (0x7fd44fa96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #17: + 0x1aa7828 (0x7fd44bf67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #18: + 0x5fd6d74 (0x7fd450496d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #19: + 0x5fd7b05 (0x7fd450497b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank13]: frame #20: + 0xe1df28 (0x7fd45ff83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank13]: frame #21: + 0x4cf144 (0x7fd45f635144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank13]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank13]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank13]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank13]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank13]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank13]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank13]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank13]: frame #41: + 0x29d90 (0x7fd47694bd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank13]: frame #42: __libc_start_main + 0x80 (0x7fd47694be40 in /lib/x86_64-linux-gnu/libc.so.6) [rank13]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank13]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-629: task 13: Exited with exit code 1 [rank16]:[W1114 14:31:26.529213820 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 16] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank16]: Traceback (most recent call last): [rank16]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank16]: main() [rank16]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank16]: dist.barrier() [rank16]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank16]: return func(*args, **kwargs) [rank16]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank16]: work = group.barrier(opts=opts) [rank16]: torch.distributed.DistBackendError: [16] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank16]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank16]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f7c15cb8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank16]: frame #1: + 0x6021e58 (0x7f7c058e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f7c058debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f7c058def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f7c058dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7c0588ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7c0588ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7c0588ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f7bb855adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank16]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f7bb85674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank16]: frame #10: + 0x1205255 (0x7f7bb856d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank16]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f7bb856e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank16]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f7bb857f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank16]: frame #13: + 0x5fc28a2 (0x7f7c058828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #14: + 0x5fcd035 (0x7f7c0588d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #15: + 0x55d8ceb (0x7f7c04e98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #16: + 0x55d6579 (0x7f7c04e96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #17: + 0x1aa7828 (0x7f7c01367828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #18: + 0x5fd6d74 (0x7f7c05896d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #19: + 0x5fd7b05 (0x7f7c05897b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank16]: frame #20: + 0xe1df28 (0x7f7c15383f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank16]: frame #21: + 0x4cf144 (0x7f7c14a35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank16]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank16]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank16]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank16]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank16]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank16]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank16]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank16]: frame #41: + 0x29d90 (0x7f7c2bcf6d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank16]: frame #42: __libc_start_main + 0x80 (0x7f7c2bcf6e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank16]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank16]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank16]:[W1114 14:31:26.147112462 ProcessGroupNCCL.cpp:1326] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) srun: error: h100-ord01-03-729: task 16: Exited with exit code 1 [rank14]:[W1114 14:31:48.913819901 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 14] using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank14]: Traceback (most recent call last): [rank14]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank14]: main() [rank14]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank14]: dist.barrier() [rank14]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank14]: return func(*args, **kwargs) [rank14]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank14]: work = group.barrier(opts=opts) [rank14]: torch.distributed.DistBackendError: [14] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank14]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank14]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f2e99f6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank14]: frame #1: + 0x6021e58 (0x7f2e89ae1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f2e89adebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f2e89adef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f2e89adfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2e89a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2e89a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f2e89a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f2e3c75adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank14]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f2e3c7674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank14]: frame #10: + 0x1205255 (0x7f2e3c76d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank14]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f2e3c76e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank14]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f2e3c77f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank14]: frame #13: + 0x5fc28a2 (0x7f2e89a828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #14: + 0x5fcd035 (0x7f2e89a8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #15: + 0x55d8ceb (0x7f2e89098ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #16: + 0x55d6579 (0x7f2e89096579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #17: + 0x1aa7828 (0x7f2e85567828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #18: + 0x5fd6d74 (0x7f2e89a96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #19: + 0x5fd7b05 (0x7f2e89a97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank14]: frame #20: + 0xe1df28 (0x7f2e99583f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank14]: frame #21: + 0x4cf144 (0x7f2e98c35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank14]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank14]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank14]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank14]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank14]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank14]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank14]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank14]: frame #41: + 0x29d90 (0x7f2eaff42d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank14]: frame #42: __libc_start_main + 0x80 (0x7f2eaff42e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank14]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank14]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-629: task 14: Exited with exit code 1 [rank24]:[W1114 14:31:53.896312479 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 24] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank24]: Traceback (most recent call last): [rank24]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank24]: main() [rank24]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank24]: dist.barrier() [rank24]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank24]: return func(*args, **kwargs) [rank24]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank24]: work = group.barrier(opts=opts) [rank24]: torch.distributed.DistBackendError: [24] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank24]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank24]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f7ec30b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank24]: frame #1: + 0x6021e58 (0x7f7eb2ce1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f7eb2cdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f7eb2cdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f7eb2cdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7eb2c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7eb2c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7eb2c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f7e6595adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank24]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f7e659674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank24]: frame #10: + 0x1205255 (0x7f7e6596d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank24]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f7e6596e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank24]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f7e6597f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank24]: frame #13: + 0x5fc28a2 (0x7f7eb2c828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #14: + 0x5fcd035 (0x7f7eb2c8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #15: + 0x55d8ceb (0x7f7eb2298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #16: + 0x55d6579 (0x7f7eb2296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #17: + 0x1aa7828 (0x7f7eae767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #18: + 0x5fd6d74 (0x7f7eb2c96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #19: + 0x5fd7b05 (0x7f7eb2c97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank24]: frame #20: + 0xe1df28 (0x7f7ec2783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank24]: frame #21: + 0x4cf144 (0x7f7ec1e35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank24]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank24]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank24]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank24]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank24]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank24]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank24]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank24]: frame #41: + 0x29d90 (0x7f7ed90e8d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank24]: frame #42: __libc_start_main + 0x80 (0x7f7ed90e8e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank24]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank24]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank18]:[W1114 14:31:54.462767284 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 18] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank18]: Traceback (most recent call last): [rank18]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank18]: main() [rank18]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank18]: dist.barrier() [rank18]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank18]: return func(*args, **kwargs) [rank18]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank18]: work = group.barrier(opts=opts) [rank18]: torch.distributed.DistBackendError: [18] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank18]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank18]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7ff7d00b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank18]: frame #1: + 0x6021e58 (0x7ff7bfce1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7ff7bfcdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7ff7bfcdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7ff7bfcdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff7bfc8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff7bfc8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7ff7bfc8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7ff77295adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank18]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7ff7729674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank18]: frame #10: + 0x1205255 (0x7ff77296d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank18]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7ff77296e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank18]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7ff77297f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank18]: frame #13: + 0x5fc28a2 (0x7ff7bfc828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #14: + 0x5fcd035 (0x7ff7bfc8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #15: + 0x55d8ceb (0x7ff7bf298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #16: + 0x55d6579 (0x7ff7bf296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #17: + 0x1aa7828 (0x7ff7bb767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #18: + 0x5fd6d74 (0x7ff7bfc96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #19: + 0x5fd7b05 (0x7ff7bfc97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank18]: frame #20: + 0xe1df28 (0x7ff7cf783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank18]: frame #21: + 0x4cf144 (0x7ff7cee35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank18]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank18]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank18]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank18]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank18]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank18]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank18]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank18]: frame #41: + 0x29d90 (0x7ff7e60e9d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank18]: frame #42: __libc_start_main + 0x80 (0x7ff7e60e9e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank18]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank18]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank24]:[W1114 14:31:54.504496592 ProcessGroupNCCL.cpp:1326] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) srun: error: h100-ord01-03-1023: task 24: Exited with exit code 1 srun: error: h100-ord01-03-729: task 18: Exited with exit code 1 [rank62]:[W1114 14:32:13.290772668 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 62] using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank62]: Traceback (most recent call last): [rank62]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank62]: main() [rank62]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank62]: dist.barrier() [rank62]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank62]: return func(*args, **kwargs) [rank62]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank62]: work = group.barrier(opts=opts) [rank62]: torch.distributed.DistBackendError: [62] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank62]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank62]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f6f77d6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank62]: frame #1: + 0x6021e58 (0x7f6f678e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f6f678debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f6f678def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f6f678dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6f6788ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6f6788ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6f6788ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f6f1a55adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank62]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f6f1a5674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank62]: frame #10: + 0x1205255 (0x7f6f1a56d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank62]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f6f1a56e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank62]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f6f1a57f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank62]: frame #13: + 0x5fc28a2 (0x7f6f678828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #14: + 0x5fcd035 (0x7f6f6788d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #15: + 0x55d8ceb (0x7f6f66e98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #16: + 0x55d6579 (0x7f6f66e96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #17: + 0x1aa7828 (0x7f6f63367828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #18: + 0x5fd6d74 (0x7f6f67896d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #19: + 0x5fd7b05 (0x7f6f67897b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank62]: frame #20: + 0xe1df28 (0x7f6f77383f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank62]: frame #21: + 0x4cf144 (0x7f6f76a35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank62]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank62]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank62]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank62]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank62]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank62]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank62]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank62]: frame #41: + 0x29d90 (0x7f6f8dddfd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank62]: frame #42: __libc_start_main + 0x80 (0x7f6f8dddfe40 in /lib/x86_64-linux-gnu/libc.so.6) [rank62]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank62]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-948: task 62: Exited with exit code 1 [rank40]:[W1114 14:32:16.371642516 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 40] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank40]: Traceback (most recent call last): [rank40]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank40]: main() [rank40]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank40]: dist.barrier() [rank40]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank40]: return func(*args, **kwargs) [rank40]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank40]: work = group.barrier(opts=opts) [rank40]: torch.distributed.DistBackendError: [40] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank40]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank40]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fa424ab8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank40]: frame #1: + 0x6021e58 (0x7fa4146e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fa4146debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fa4146def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fa4146dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa41468ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa41468ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fa41468ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fa3c735adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank40]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fa3c73674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank40]: frame #10: + 0x1205255 (0x7fa3c736d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank40]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fa3c736e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank40]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fa3c737f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank40]: frame #13: + 0x5fc28a2 (0x7fa4146828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #14: + 0x5fcd035 (0x7fa41468d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #15: + 0x55d8ceb (0x7fa413c98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #16: + 0x55d6579 (0x7fa413c96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #17: + 0x1aa7828 (0x7fa410167828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #18: + 0x5fd6d74 (0x7fa414696d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #19: + 0x5fd7b05 (0x7fa414697b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank40]: frame #20: + 0xe1df28 (0x7fa424183f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank40]: frame #21: + 0x4cf144 (0x7fa423835144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank40]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank40]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank40]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank40]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank40]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank40]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank40]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank40]: frame #41: + 0x29d90 (0x7fa43aa9cd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank40]: frame #42: __libc_start_main + 0x80 (0x7fa43aa9ce40 in /lib/x86_64-linux-gnu/libc.so.6) [rank40]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank40]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank40]:[W1114 14:32:17.043812039 ProcessGroupNCCL.cpp:1326] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) srun: error: h100-ord01-03-1017: task 40: Exited with exit code 1 [rank37]:[W1114 14:32:49.697308954 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 37] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank37]: Traceback (most recent call last): [rank37]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank37]: main() [rank37]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank37]: dist.barrier() [rank37]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank37]: return func(*args, **kwargs) [rank37]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank37]: work = group.barrier(opts=opts) [rank37]: torch.distributed.DistBackendError: [37] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank37]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank37]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f7acc6b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank37]: frame #1: + 0x6021e58 (0x7f7abc2e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f7abc2debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f7abc2def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f7abc2dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7abc28ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7abc28ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7abc28ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f7a6ef5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank37]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f7a6ef674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank37]: frame #10: + 0x1205255 (0x7f7a6ef6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank37]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f7a6ef6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank37]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f7a6ef7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank37]: frame #13: + 0x5fc28a2 (0x7f7abc2828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #14: + 0x5fcd035 (0x7f7abc28d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #15: + 0x55d8ceb (0x7f7abb898ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #16: + 0x55d6579 (0x7f7abb896579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #17: + 0x1aa7828 (0x7f7ab7d67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #18: + 0x5fd6d74 (0x7f7abc296d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #19: + 0x5fd7b05 (0x7f7abc297b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank37]: frame #20: + 0xe1df28 (0x7f7acbd83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank37]: frame #21: + 0x4cf144 (0x7f7acb435144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank37]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank37]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank37]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank37]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank37]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank37]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank37]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank37]: frame #41: + 0x29d90 (0x7f7ae270bd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank37]: frame #42: __libc_start_main + 0x80 (0x7f7ae270be40 in /lib/x86_64-linux-gnu/libc.so.6) [rank37]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank37]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-489: task 37: Exited with exit code 1 [rank42]:[W1114 14:32:57.825171972 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 42] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank42]: Traceback (most recent call last): [rank42]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank42]: main() [rank42]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank42]: dist.barrier() [rank42]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank42]: return func(*args, **kwargs) [rank42]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank42]: work = group.barrier(opts=opts) [rank42]: torch.distributed.DistBackendError: [42] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank42]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank42]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd03896bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank42]: frame #1: + 0x6021e58 (0x7fd0284e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fd0284debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fd0284def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fd0284dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd02848ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd02848ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd02848ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fcfdb15adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank42]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fcfdb1674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank42]: frame #10: + 0x1205255 (0x7fcfdb16d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank42]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fcfdb16e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank42]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fcfdb17f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank42]: frame #13: + 0x5fc28a2 (0x7fd0284828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #14: + 0x5fcd035 (0x7fd02848d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #15: + 0x55d8ceb (0x7fd027a98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #16: + 0x55d6579 (0x7fd027a96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #17: + 0x1aa7828 (0x7fd023f67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #18: + 0x5fd6d74 (0x7fd028496d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #19: + 0x5fd7b05 (0x7fd028497b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank42]: frame #20: + 0xe1df28 (0x7fd037f83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank42]: frame #21: + 0x4cf144 (0x7fd037635144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank42]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank42]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank42]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank42]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank42]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank42]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank42]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank42]: frame #41: + 0x29d90 (0x7fd04e92ed90 in /lib/x86_64-linux-gnu/libc.so.6) [rank42]: frame #42: __libc_start_main + 0x80 (0x7fd04e92ee40 in /lib/x86_64-linux-gnu/libc.so.6) [rank42]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank42]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-1017: task 42: Exited with exit code 1 [rank44]:[W1114 14:33:05.058692315 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 44] using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank44]: Traceback (most recent call last): [rank44]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank44]: main() [rank44]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank44]: dist.barrier() [rank44]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank44]: return func(*args, **kwargs) [rank44]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank44]: work = group.barrier(opts=opts) [rank44]: torch.distributed.DistBackendError: [44] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank44]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank44]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fcf276b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank44]: frame #1: + 0x6021e58 (0x7fcf172e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fcf172debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fcf172def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fcf172dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fcf1728ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fcf1728ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fcf1728ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fcec9f5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank44]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fcec9f674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank44]: frame #10: + 0x1205255 (0x7fcec9f6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank44]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fcec9f6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank44]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fcec9f7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank44]: frame #13: + 0x5fc28a2 (0x7fcf172828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #14: + 0x5fcd035 (0x7fcf1728d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #15: + 0x55d8ceb (0x7fcf16898ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #16: + 0x55d6579 (0x7fcf16896579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #17: + 0x1aa7828 (0x7fcf12d67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #18: + 0x5fd6d74 (0x7fcf17296d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #19: + 0x5fd7b05 (0x7fcf17297b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank44]: frame #20: + 0xe1df28 (0x7fcf26d83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank44]: frame #21: + 0x4cf144 (0x7fcf26435144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank44]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank44]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank44]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank44]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank44]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank44]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank44]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank44]: frame #41: + 0x29d90 (0x7fcf3d6a8d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank44]: frame #42: __libc_start_main + 0x80 (0x7fcf3d6a8e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank44]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank44]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-1017: task 44: Exited with exit code 1 [rank60]:[W1114 14:33:16.489692039 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 60] using GPU 4 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank60]: Traceback (most recent call last): [rank60]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank60]: main() [rank60]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank60]: dist.barrier() [rank60]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank60]: return func(*args, **kwargs) [rank60]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank60]: work = group.barrier(opts=opts) [rank60]: torch.distributed.DistBackendError: [60] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank60]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank60]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fb1394b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank60]: frame #1: + 0x6021e58 (0x7fb1290e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fb1290debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fb1290def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fb1290dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb12908ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb12908ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb12908ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fb0dbd5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank60]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fb0dbd674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank60]: frame #10: + 0x1205255 (0x7fb0dbd6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank60]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fb0dbd6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank60]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fb0dbd7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank60]: frame #13: + 0x5fc28a2 (0x7fb1290828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #14: + 0x5fcd035 (0x7fb12908d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #15: + 0x55d8ceb (0x7fb128698ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #16: + 0x55d6579 (0x7fb128696579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #17: + 0x1aa7828 (0x7fb124b67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #18: + 0x5fd6d74 (0x7fb129096d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #19: + 0x5fd7b05 (0x7fb129097b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank60]: frame #20: + 0xe1df28 (0x7fb138b83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank60]: frame #21: + 0x4cf144 (0x7fb138235144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank60]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank60]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank60]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank60]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank60]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank60]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank60]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank60]: frame #41: + 0x29d90 (0x7fb14f4e4d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank60]: frame #42: __libc_start_main + 0x80 (0x7fb14f4e4e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank60]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank60]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-948: task 60: Exited with exit code 1 [rank53]:[W1114 14:33:19.841218174 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 53] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank53]: Traceback (most recent call last): [rank53]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank53]: main() [rank53]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank53]: dist.barrier() [rank53]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank53]: return func(*args, **kwargs) [rank53]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank53]: work = group.barrier(opts=opts) [rank53]: torch.distributed.DistBackendError: [53] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank53]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank53]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f6c77d6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank53]: frame #1: + 0x6021e58 (0x7f6c678e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f6c678debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f6c678def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f6c678dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6c6788ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6c6788ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6c6788ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f6c1a55adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank53]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f6c1a5674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank53]: frame #10: + 0x1205255 (0x7f6c1a56d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank53]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f6c1a56e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank53]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f6c1a57f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank53]: frame #13: + 0x5fc28a2 (0x7f6c678828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #14: + 0x5fcd035 (0x7f6c6788d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #15: + 0x55d8ceb (0x7f6c66e98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #16: + 0x55d6579 (0x7f6c66e96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #17: + 0x1aa7828 (0x7f6c63367828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #18: + 0x5fd6d74 (0x7f6c67896d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #19: + 0x5fd7b05 (0x7f6c67897b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank53]: frame #20: + 0xe1df28 (0x7f6c77383f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank53]: frame #21: + 0x4cf144 (0x7f6c76a35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank53]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank53]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank53]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank53]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank53]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank53]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank53]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank53]: frame #41: + 0x29d90 (0x7f6c8dd58d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank53]: frame #42: __libc_start_main + 0x80 (0x7f6c8dd58e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank53]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank53]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-313: task 53: Exited with exit code 1 [rank8]:[W1114 14:33:23.384340640 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 8] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank8]: Traceback (most recent call last): [rank8]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank8]: main() [rank8]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank8]: dist.barrier() [rank8]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank8]: return func(*args, **kwargs) [rank8]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank8]: work = group.barrier(opts=opts) [rank8]: torch.distributed.DistBackendError: [8] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank8]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank8]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd38d6b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank8]: frame #1: + 0x6021e58 (0x7fd37d2e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fd37d2debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fd37d2def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fd37d2dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd37d28ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd37d28ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd37d28ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fd32ff5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank8]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fd32ff674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank8]: frame #10: + 0x1205255 (0x7fd32ff6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank8]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fd32ff6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank8]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fd32ff7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank8]: frame #13: + 0x5fc28a2 (0x7fd37d2828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #14: + 0x5fcd035 (0x7fd37d28d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #15: + 0x55d8ceb (0x7fd37c898ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #16: + 0x55d6579 (0x7fd37c896579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #17: + 0x1aa7828 (0x7fd378d67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #18: + 0x5fd6d74 (0x7fd37d296d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #19: + 0x5fd7b05 (0x7fd37d297b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank8]: frame #20: + 0xe1df28 (0x7fd38cd83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank8]: frame #21: + 0x4cf144 (0x7fd38c435144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank8]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank8]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank8]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank8]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank8]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank8]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank8]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank8]: frame #41: + 0x29d90 (0x7fd3a3688d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank8]: frame #42: __libc_start_main + 0x80 (0x7fd3a3688e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank8]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank8]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank8]:[W1114 14:33:24.028015404 ProcessGroupNCCL.cpp:1326] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) srun: error: h100-ord01-03-629: task 8: Exited with exit code 1 [rank10]:[W1114 14:33:26.717315858 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 10] using GPU 2 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank10]: Traceback (most recent call last): [rank10]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank10]: main() [rank10]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank10]: dist.barrier() [rank10]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank10]: return func(*args, **kwargs) [rank10]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank10]: work = group.barrier(opts=opts) [rank10]: torch.distributed.DistBackendError: [10] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank10]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank10]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fb080b6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank10]: frame #1: + 0x6021e58 (0x7fb0706e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fb0706debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fb0706def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fb0706dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb07068ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb07068ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fb07068ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fb02335adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank10]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fb0233674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank10]: frame #10: + 0x1205255 (0x7fb02336d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank10]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fb02336e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank10]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fb02337f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank10]: frame #13: + 0x5fc28a2 (0x7fb0706828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #14: + 0x5fcd035 (0x7fb07068d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #15: + 0x55d8ceb (0x7fb06fc98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #16: + 0x55d6579 (0x7fb06fc96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #17: + 0x1aa7828 (0x7fb06c167828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #18: + 0x5fd6d74 (0x7fb070696d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #19: + 0x5fd7b05 (0x7fb070697b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank10]: frame #20: + 0xe1df28 (0x7fb080183f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank10]: frame #21: + 0x4cf144 (0x7fb07f835144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank10]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank10]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank10]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank10]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank10]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank10]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank10]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank10]: frame #41: + 0x29d90 (0x7fb096bd2d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank10]: frame #42: __libc_start_main + 0x80 (0x7fb096bd2e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank10]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank10]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-629: task 10: Exited with exit code 1 [rank33]:[W1114 14:33:31.308498301 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 33] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank33]: Traceback (most recent call last): [rank33]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank33]: main() [rank33]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank33]: dist.barrier() [rank33]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank33]: return func(*args, **kwargs) [rank33]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank33]: work = group.barrier(opts=opts) [rank33]: torch.distributed.DistBackendError: [33] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank33]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank33]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f96f096bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank33]: frame #1: + 0x6021e58 (0x7f96e04e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f96e04debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f96e04def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f96e04dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f96e048ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f96e048ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f96e048ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f969315adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank33]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f96931674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank33]: frame #10: + 0x1205255 (0x7f969316d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank33]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f969316e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank33]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f969317f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank33]: frame #13: + 0x5fc28a2 (0x7f96e04828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #14: + 0x5fcd035 (0x7f96e048d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #15: + 0x55d8ceb (0x7f96dfa98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #16: + 0x55d6579 (0x7f96dfa96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #17: + 0x1aa7828 (0x7f96dbf67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #18: + 0x5fd6d74 (0x7f96e0496d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #19: + 0x5fd7b05 (0x7f96e0497b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank33]: frame #20: + 0xe1df28 (0x7f96eff83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank33]: frame #21: + 0x4cf144 (0x7f96ef635144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank33]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank33]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank33]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank33]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank33]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank33]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank33]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank33]: frame #41: + 0x29d90 (0x7f97069e8d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank33]: frame #42: __libc_start_main + 0x80 (0x7f97069e8e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank33]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank33]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-489: task 33: Exited with exit code 1 [rank30]:[W1114 14:34:07.210058557 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 30] using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank30]: Traceback (most recent call last): [rank30]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank30]: main() [rank30]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank30]: dist.barrier() [rank30]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank30]: return func(*args, **kwargs) [rank30]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank30]: work = group.barrier(opts=opts) [rank30]: torch.distributed.DistBackendError: [30] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank30]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank30]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f849a56bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank30]: frame #1: + 0x6021e58 (0x7f848a0e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f848a0debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f848a0def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f848a0dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f848a08ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f848a08ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f848a08ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f843cd5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank30]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f843cd674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank30]: frame #10: + 0x1205255 (0x7f843cd6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank30]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f843cd6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank30]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f843cd7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank30]: frame #13: + 0x5fc28a2 (0x7f848a0828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #14: + 0x5fcd035 (0x7f848a08d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #15: + 0x55d8ceb (0x7f8489698ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #16: + 0x55d6579 (0x7f8489696579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #17: + 0x1aa7828 (0x7f8485b67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #18: + 0x5fd6d74 (0x7f848a096d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #19: + 0x5fd7b05 (0x7f848a097b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank30]: frame #20: + 0xe1df28 (0x7f8499b83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank30]: frame #21: + 0x4cf144 (0x7f8499235144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank30]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank30]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank30]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank30]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank30]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank30]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank30]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank30]: frame #41: + 0x29d90 (0x7f84b05eed90 in /lib/x86_64-linux-gnu/libc.so.6) [rank30]: frame #42: __libc_start_main + 0x80 (0x7f84b05eee40 in /lib/x86_64-linux-gnu/libc.so.6) [rank30]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank30]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-1023: task 30: Exited with exit code 1 [rank46]:[W1114 14:34:18.499723426 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 46] using GPU 6 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank46]: Traceback (most recent call last): [rank46]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank46]: main() [rank46]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank46]: dist.barrier() [rank46]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank46]: return func(*args, **kwargs) [rank46]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank46]: work = group.barrier(opts=opts) [rank46]: torch.distributed.DistBackendError: [46] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank46]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank46]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f14b9b6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank46]: frame #1: + 0x6021e58 (0x7f14a96e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f14a96debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f14a96def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f14a96dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f14a968ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f14a968ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f14a968ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f145c35adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank46]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f145c3674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank46]: frame #10: + 0x1205255 (0x7f145c36d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank46]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f145c36e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank46]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f145c37f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank46]: frame #13: + 0x5fc28a2 (0x7f14a96828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #14: + 0x5fcd035 (0x7f14a968d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #15: + 0x55d8ceb (0x7f14a8c98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #16: + 0x55d6579 (0x7f14a8c96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #17: + 0x1aa7828 (0x7f14a5167828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #18: + 0x5fd6d74 (0x7f14a9696d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #19: + 0x5fd7b05 (0x7f14a9697b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank46]: frame #20: + 0xe1df28 (0x7f14b9183f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank46]: frame #21: + 0x4cf144 (0x7f14b8835144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank46]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank46]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank46]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank46]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank46]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank46]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank46]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank46]: frame #41: + 0x29d90 (0x7f14cfbf7d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank46]: frame #42: __libc_start_main + 0x80 (0x7f14cfbf7e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank46]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank46]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-1017: task 46: Exited with exit code 1 [rank35]:[W1114 14:34:53.945741311 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 35] using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank35]: Traceback (most recent call last): [rank35]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank35]: main() [rank35]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank35]: dist.barrier() [rank35]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank35]: return func(*args, **kwargs) [rank35]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank35]: work = group.barrier(opts=opts) [rank35]: torch.distributed.DistBackendError: [35] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank35]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank35]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f761216bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank35]: frame #1: + 0x6021e58 (0x7f7601ce1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f7601cdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f7601cdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f7601cdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7601c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7601c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7601c8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f75b495adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank35]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f75b49674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank35]: frame #10: + 0x1205255 (0x7f75b496d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank35]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f75b496e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank35]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f75b497f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank35]: frame #13: + 0x5fc28a2 (0x7f7601c828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #14: + 0x5fcd035 (0x7f7601c8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #15: + 0x55d8ceb (0x7f7601298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #16: + 0x55d6579 (0x7f7601296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #17: + 0x1aa7828 (0x7f75fd767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #18: + 0x5fd6d74 (0x7f7601c96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #19: + 0x5fd7b05 (0x7f7601c97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank35]: frame #20: + 0xe1df28 (0x7f7611783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank35]: frame #21: + 0x4cf144 (0x7f7610e35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank35]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank35]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank35]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank35]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank35]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank35]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank35]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank35]: frame #41: + 0x29d90 (0x7f762815fd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank35]: frame #42: __libc_start_main + 0x80 (0x7f762815fe40 in /lib/x86_64-linux-gnu/libc.so.6) [rank35]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank35]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-489: task 35: Exited with exit code 1 [rank19]:[W1114 14:35:18.014738334 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 19] using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank19]: Traceback (most recent call last): [rank19]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank19]: main() [rank19]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank19]: dist.barrier() [rank19]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank19]: return func(*args, **kwargs) [rank19]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank19]: work = group.barrier(opts=opts) [rank19]: torch.distributed.DistBackendError: [19] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank19]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank19]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fca37b6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank19]: frame #1: + 0x6021e58 (0x7fca276e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fca276debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fca276def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fca276dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fca2768ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fca2768ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fca2768ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fc9da35adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank19]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fc9da3674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank19]: frame #10: + 0x1205255 (0x7fc9da36d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank19]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fc9da36e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank19]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fc9da37f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank19]: frame #13: + 0x5fc28a2 (0x7fca276828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #14: + 0x5fcd035 (0x7fca2768d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #15: + 0x55d8ceb (0x7fca26c98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #16: + 0x55d6579 (0x7fca26c96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #17: + 0x1aa7828 (0x7fca23167828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #18: + 0x5fd6d74 (0x7fca27696d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #19: + 0x5fd7b05 (0x7fca27697b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank19]: frame #20: + 0xe1df28 (0x7fca37183f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank19]: frame #21: + 0x4cf144 (0x7fca36835144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank19]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank19]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank19]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank19]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank19]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank19]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank19]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank19]: frame #41: + 0x29d90 (0x7fca4dbbbd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank19]: frame #42: __libc_start_main + 0x80 (0x7fca4dbbbe40 in /lib/x86_64-linux-gnu/libc.so.6) [rank19]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank19]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-729: task 19: Exited with exit code 1 [rank43]:[W1114 14:35:22.850703055 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 43] using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank43]: Traceback (most recent call last): [rank43]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank43]: main() [rank43]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank43]: dist.barrier() [rank43]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank43]: return func(*args, **kwargs) [rank43]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank43]: work = group.barrier(opts=opts) [rank43]: torch.distributed.DistBackendError: [43] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank43]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank43]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f5fccad3cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank43]: frame #1: + 0x6021e58 (0x7f601b0e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f601b0debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f601b0def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f601b0dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f601b08ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f601b08ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f601b08ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f5fcdd5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank43]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f5fcdd674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank43]: frame #10: + 0x1205255 (0x7f5fcdd6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank43]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f5fcdd6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank43]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f5fcdd7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank43]: frame #13: + 0x5fc28a2 (0x7f601b0828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #14: + 0x5fcd035 (0x7f601b08d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #15: + 0x55d8ceb (0x7f601a698ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #16: + 0x55d6579 (0x7f601a696579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #17: + 0x1aa7828 (0x7f6016b67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #18: + 0x5fd6d74 (0x7f601b096d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #19: + 0x5fd7b05 (0x7f601b097b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank43]: frame #20: + 0xe1df28 (0x7f602ab83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank43]: frame #21: + 0x4cf144 (0x7f602a235144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank43]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank43]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank43]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank43]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank43]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank43]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank43]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank43]: frame #41: + 0x29d90 (0x7f6041457d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank43]: frame #42: __libc_start_main + 0x80 (0x7f6041457e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank43]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank43]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-1017: task 43: Exited with exit code 1 [rank23]:[W1114 14:35:37.174666562 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 23] using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank23]: Traceback (most recent call last): [rank23]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank23]: main() [rank23]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank23]: dist.barrier() [rank23]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank23]: return func(*args, **kwargs) [rank23]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank23]: work = group.barrier(opts=opts) [rank23]: torch.distributed.DistBackendError: [23] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank23]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank23]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f7ec636bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank23]: frame #1: + 0x6021e58 (0x7f7eb5ee1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f7eb5edebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f7eb5edef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f7eb5edfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7eb5e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7eb5e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7eb5e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f7e68b5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank23]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f7e68b674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank23]: frame #10: + 0x1205255 (0x7f7e68b6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank23]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f7e68b6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank23]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f7e68b7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank23]: frame #13: + 0x5fc28a2 (0x7f7eb5e828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #14: + 0x5fcd035 (0x7f7eb5e8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #15: + 0x55d8ceb (0x7f7eb5498ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #16: + 0x55d6579 (0x7f7eb5496579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #17: + 0x1aa7828 (0x7f7eb1967828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #18: + 0x5fd6d74 (0x7f7eb5e96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #19: + 0x5fd7b05 (0x7f7eb5e97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank23]: frame #20: + 0xe1df28 (0x7f7ec5983f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank23]: frame #21: + 0x4cf144 (0x7f7ec5035144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank23]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank23]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank23]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank23]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank23]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank23]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank23]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank23]: frame #41: + 0x29d90 (0x7f7edc3f3d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank23]: frame #42: __libc_start_main + 0x80 (0x7f7edc3f3e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank23]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank23]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-729: task 23: Exited with exit code 1 [rank31]:[W1114 14:35:42.204259712 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 31] using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank31]: Traceback (most recent call last): [rank31]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank31]: main() [rank31]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank31]: dist.barrier() [rank31]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank31]: return func(*args, **kwargs) [rank31]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank31]: work = group.barrier(opts=opts) [rank31]: torch.distributed.DistBackendError: [31] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank31]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank31]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f690f0b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank31]: frame #1: + 0x6021e58 (0x7f68fece1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f68fecdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f68fecdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f68fecdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f68fec8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f68fec8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f68fec8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f68b195adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank31]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f68b19674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank31]: frame #10: + 0x1205255 (0x7f68b196d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank31]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f68b196e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank31]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f68b197f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank31]: frame #13: + 0x5fc28a2 (0x7f68fec828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #14: + 0x5fcd035 (0x7f68fec8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #15: + 0x55d8ceb (0x7f68fe298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #16: + 0x55d6579 (0x7f68fe296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #17: + 0x1aa7828 (0x7f68fa767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #18: + 0x5fd6d74 (0x7f68fec96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #19: + 0x5fd7b05 (0x7f68fec97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank31]: frame #20: + 0xe1df28 (0x7f690e783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank31]: frame #21: + 0x4cf144 (0x7f690de35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank31]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank31]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank31]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank31]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank31]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank31]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank31]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank31]: frame #41: + 0x29d90 (0x7f6925082d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank31]: frame #42: __libc_start_main + 0x80 (0x7f6925082e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank31]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank31]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-1023: task 31: Exited with exit code 1 [rank55]:[W1114 14:36:14.256567302 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 55] using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank55]: Traceback (most recent call last): [rank55]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank55]: main() [rank55]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank55]: dist.barrier() [rank55]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank55]: return func(*args, **kwargs) [rank55]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank55]: work = group.barrier(opts=opts) [rank55]: torch.distributed.DistBackendError: [55] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank55]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank55]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fc853cb8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank55]: frame #1: + 0x6021e58 (0x7fc8438e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fc8438debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fc8438def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fc8438dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc84388ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc84388ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fc84388ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fc7f655adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank55]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fc7f65674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank55]: frame #10: + 0x1205255 (0x7fc7f656d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank55]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fc7f656e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank55]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fc7f657f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank55]: frame #13: + 0x5fc28a2 (0x7fc8438828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #14: + 0x5fcd035 (0x7fc84388d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #15: + 0x55d8ceb (0x7fc842e98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #16: + 0x55d6579 (0x7fc842e96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #17: + 0x1aa7828 (0x7fc83f367828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #18: + 0x5fd6d74 (0x7fc843896d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #19: + 0x5fd7b05 (0x7fc843897b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank55]: frame #20: + 0xe1df28 (0x7fc853383f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank55]: frame #21: + 0x4cf144 (0x7fc852a35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank55]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank55]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank55]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank55]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank55]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank55]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank55]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank55]: frame #41: + 0x29d90 (0x7fc869cb8d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank55]: frame #42: __libc_start_main + 0x80 (0x7fc869cb8e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank55]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank55]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-313: task 55: Exited with exit code 1 [rank61]:[W1114 14:36:28.967787967 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 61] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank61]: Traceback (most recent call last): [rank61]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank61]: main() [rank61]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank61]: dist.barrier() [rank61]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank61]: return func(*args, **kwargs) [rank61]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank61]: work = group.barrier(opts=opts) [rank61]: torch.distributed.DistBackendError: [61] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank61]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank61]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f3c6356bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank61]: frame #1: + 0x6021e58 (0x7f3c530e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f3c530debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f3c530def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f3c530dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3c5308ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3c5308ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3c5308ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f3c05d5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank61]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f3c05d674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank61]: frame #10: + 0x1205255 (0x7f3c05d6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank61]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f3c05d6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank61]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f3c05d7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank61]: frame #13: + 0x5fc28a2 (0x7f3c530828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #14: + 0x5fcd035 (0x7f3c5308d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #15: + 0x55d8ceb (0x7f3c52698ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #16: + 0x55d6579 (0x7f3c52696579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #17: + 0x1aa7828 (0x7f3c4eb67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #18: + 0x5fd6d74 (0x7f3c53096d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #19: + 0x5fd7b05 (0x7f3c53097b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank61]: frame #20: + 0xe1df28 (0x7f3c62b83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank61]: frame #21: + 0x4cf144 (0x7f3c62235144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank61]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank61]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank61]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank61]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank61]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank61]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank61]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank61]: frame #41: + 0x29d90 (0x7f3c7956bd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank61]: frame #42: __libc_start_main + 0x80 (0x7f3c7956be40 in /lib/x86_64-linux-gnu/libc.so.6) [rank61]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank61]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-948: task 61: Exited with exit code 1 [rank21]:[W1114 14:36:41.891367646 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 21] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank21]: Traceback (most recent call last): [rank21]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank21]: main() [rank21]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank21]: dist.barrier() [rank21]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank21]: return func(*args, **kwargs) [rank21]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank21]: work = group.barrier(opts=opts) [rank21]: torch.distributed.DistBackendError: [21] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank21]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank21]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd8bcd6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank21]: frame #1: + 0x6021e58 (0x7fd8ac8e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fd8ac8debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fd8ac8def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fd8ac8dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd8ac88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd8ac88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd8ac88ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fd85f55adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank21]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fd85f5674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank21]: frame #10: + 0x1205255 (0x7fd85f56d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank21]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fd85f56e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank21]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fd85f57f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank21]: frame #13: + 0x5fc28a2 (0x7fd8ac8828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #14: + 0x5fcd035 (0x7fd8ac88d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #15: + 0x55d8ceb (0x7fd8abe98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #16: + 0x55d6579 (0x7fd8abe96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #17: + 0x1aa7828 (0x7fd8a8367828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #18: + 0x5fd6d74 (0x7fd8ac896d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #19: + 0x5fd7b05 (0x7fd8ac897b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank21]: frame #20: + 0xe1df28 (0x7fd8bc383f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank21]: frame #21: + 0x4cf144 (0x7fd8bba35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank21]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank21]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank21]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank21]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank21]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank21]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank21]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank21]: frame #41: + 0x29d90 (0x7fd8d2dfed90 in /lib/x86_64-linux-gnu/libc.so.6) [rank21]: frame #42: __libc_start_main + 0x80 (0x7fd8d2dfee40 in /lib/x86_64-linux-gnu/libc.so.6) [rank21]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank21]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-729: task 21: Exited with exit code 1 [rank27]:[W1114 14:37:15.405045198 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 27] using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank27]: Traceback (most recent call last): [rank27]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank27]: main() [rank27]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank27]: dist.barrier() [rank27]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank27]: return func(*args, **kwargs) [rank27]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank27]: work = group.barrier(opts=opts) [rank27]: torch.distributed.DistBackendError: [27] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank27]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank27]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fae3976bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank27]: frame #1: + 0x6021e58 (0x7fae292e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fae292debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fae292def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fae292dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fae2928ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fae2928ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fae2928ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7faddbf5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank27]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7faddbf674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank27]: frame #10: + 0x1205255 (0x7faddbf6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank27]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7faddbf6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank27]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7faddbf7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank27]: frame #13: + 0x5fc28a2 (0x7fae292828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #14: + 0x5fcd035 (0x7fae2928d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #15: + 0x55d8ceb (0x7fae28898ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #16: + 0x55d6579 (0x7fae28896579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #17: + 0x1aa7828 (0x7fae24d67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #18: + 0x5fd6d74 (0x7fae29296d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #19: + 0x5fd7b05 (0x7fae29297b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank27]: frame #20: + 0xe1df28 (0x7fae38d83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank27]: frame #21: + 0x4cf144 (0x7fae38435144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank27]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank27]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank27]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank27]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank27]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank27]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank27]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank27]: frame #41: + 0x29d90 (0x7fae4f78cd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank27]: frame #42: __libc_start_main + 0x80 (0x7fae4f78ce40 in /lib/x86_64-linux-gnu/libc.so.6) [rank27]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank27]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-1023: task 27: Exited with exit code 1 [rank17]:[W1114 14:37:23.525435181 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 17] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank17]: Traceback (most recent call last): [rank17]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank17]: main() [rank17]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank17]: dist.barrier() [rank17]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank17]: return func(*args, **kwargs) [rank17]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank17]: work = group.barrier(opts=opts) [rank17]: torch.distributed.DistBackendError: [17] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank17]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank17]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f295756bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank17]: frame #1: + 0x6021e58 (0x7f29470e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f29470debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f29470def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f29470dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f294708ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f294708ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f294708ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f28f9d5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank17]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f28f9d674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank17]: frame #10: + 0x1205255 (0x7f28f9d6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank17]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f28f9d6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank17]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f28f9d7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank17]: frame #13: + 0x5fc28a2 (0x7f29470828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #14: + 0x5fcd035 (0x7f294708d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #15: + 0x55d8ceb (0x7f2946698ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #16: + 0x55d6579 (0x7f2946696579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #17: + 0x1aa7828 (0x7f2942b67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #18: + 0x5fd6d74 (0x7f2947096d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #19: + 0x5fd7b05 (0x7f2947097b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank17]: frame #20: + 0xe1df28 (0x7f2956b83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank17]: frame #21: + 0x4cf144 (0x7f2956235144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank17]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank17]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank17]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank17]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank17]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank17]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank17]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank17]: frame #41: + 0x29d90 (0x7f296d5ccd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank17]: frame #42: __libc_start_main + 0x80 (0x7f296d5cce40 in /lib/x86_64-linux-gnu/libc.so.6) [rank17]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank17]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-729: task 17: Exited with exit code 1 [rank1]:[W1114 14:37:39.368736467 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank1]: Traceback (most recent call last): [rank1]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank1]: main() [rank1]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank1]: dist.barrier() [rank1]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank1]: return func(*args, **kwargs) [rank1]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank1]: work = group.barrier(opts=opts) [rank1]: torch.distributed.DistBackendError: [1] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank1]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank1]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f99f82b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank1]: frame #1: + 0x6021e58 (0x7f99e7ee1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f99e7edebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f99e7edef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f99e7edfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f99e7e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f99e7e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f99e7e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f999ab5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank1]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f999ab674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank1]: frame #10: + 0x1205255 (0x7f999ab6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank1]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f999ab6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank1]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f999ab7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank1]: frame #13: + 0x5fc28a2 (0x7f99e7e828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #14: + 0x5fcd035 (0x7f99e7e8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #15: + 0x55d8ceb (0x7f99e7498ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #16: + 0x55d6579 (0x7f99e7496579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #17: + 0x1aa7828 (0x7f99e3967828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #18: + 0x5fd6d74 (0x7f99e7e96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #19: + 0x5fd7b05 (0x7f99e7e97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank1]: frame #20: + 0xe1df28 (0x7f99f7983f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank1]: frame #21: + 0x4cf144 (0x7f99f7035144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank1]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank1]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank1]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank1]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank1]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank1]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank1]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank1]: frame #41: + 0x29d90 (0x7f9a0e2a6d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank1]: frame #42: __libc_start_main + 0x80 (0x7f9a0e2a6e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank1]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank1]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-801: task 1: Exited with exit code 1 [rank59]:[W1114 14:37:42.756214600 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 59] using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank59]: Traceback (most recent call last): [rank59]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank59]: main() [rank59]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank59]: dist.barrier() [rank59]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank59]: return func(*args, **kwargs) [rank59]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank59]: work = group.barrier(opts=opts) [rank59]: torch.distributed.DistBackendError: [59] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank59]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank59]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f6aff6b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank59]: frame #1: + 0x6021e58 (0x7f6aef2e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f6aef2debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f6aef2def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f6aef2dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6aef28ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6aef28ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6aef28ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f6aa1f5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank59]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f6aa1f674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank59]: frame #10: + 0x1205255 (0x7f6aa1f6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank59]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f6aa1f6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank59]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f6aa1f7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank59]: frame #13: + 0x5fc28a2 (0x7f6aef2828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #14: + 0x5fcd035 (0x7f6aef28d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #15: + 0x55d8ceb (0x7f6aee898ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #16: + 0x55d6579 (0x7f6aee896579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #17: + 0x1aa7828 (0x7f6aead67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #18: + 0x5fd6d74 (0x7f6aef296d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #19: + 0x5fd7b05 (0x7f6aef297b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank59]: frame #20: + 0xe1df28 (0x7f6afed83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank59]: frame #21: + 0x4cf144 (0x7f6afe435144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank59]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank59]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank59]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank59]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank59]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank59]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank59]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank59]: frame #41: + 0x29d90 (0x7f6b156f5d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank59]: frame #42: __libc_start_main + 0x80 (0x7f6b156f5e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank59]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank59]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-948: task 59: Exited with exit code 1 [rank29]:[W1114 14:37:45.890997014 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 29] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank29]: Traceback (most recent call last): [rank29]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank29]: main() [rank29]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank29]: dist.barrier() [rank29]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank29]: return func(*args, **kwargs) [rank29]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank29]: work = group.barrier(opts=opts) [rank29]: torch.distributed.DistBackendError: [29] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank29]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank29]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f3084f6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank29]: frame #1: + 0x6021e58 (0x7f3074ae1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f3074adebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f3074adef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f3074adfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3074a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3074a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f3074a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f302775adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank29]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f30277674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank29]: frame #10: + 0x1205255 (0x7f302776d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank29]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f302776e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank29]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f302777f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank29]: frame #13: + 0x5fc28a2 (0x7f3074a828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #14: + 0x5fcd035 (0x7f3074a8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #15: + 0x55d8ceb (0x7f3074098ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #16: + 0x55d6579 (0x7f3074096579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #17: + 0x1aa7828 (0x7f3070567828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #18: + 0x5fd6d74 (0x7f3074a96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #19: + 0x5fd7b05 (0x7f3074a97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank29]: frame #20: + 0xe1df28 (0x7f3084583f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank29]: frame #21: + 0x4cf144 (0x7f3083c35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank29]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank29]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank29]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank29]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank29]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank29]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank29]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank29]: frame #41: + 0x29d90 (0x7f309b017d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank29]: frame #42: __libc_start_main + 0x80 (0x7f309b017e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank29]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank29]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-1023: task 29: Exited with exit code 1 [rank25]:[W1114 14:39:03.705003078 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 25] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank25]: Traceback (most recent call last): [rank25]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank25]: main() [rank25]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank25]: dist.barrier() [rank25]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank25]: return func(*args, **kwargs) [rank25]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank25]: work = group.barrier(opts=opts) [rank25]: torch.distributed.DistBackendError: [25] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank25]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank25]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f6cf2b6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank25]: frame #1: + 0x6021e58 (0x7f6ce26e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f6ce26debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f6ce26def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f6ce26dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6ce268ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6ce268ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6ce268ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f6c9535adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank25]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f6c953674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank25]: frame #10: + 0x1205255 (0x7f6c9536d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank25]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f6c9536e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank25]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f6c9537f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank25]: frame #13: + 0x5fc28a2 (0x7f6ce26828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #14: + 0x5fcd035 (0x7f6ce268d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #15: + 0x55d8ceb (0x7f6ce1c98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #16: + 0x55d6579 (0x7f6ce1c96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #17: + 0x1aa7828 (0x7f6cde167828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #18: + 0x5fd6d74 (0x7f6ce2696d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #19: + 0x5fd7b05 (0x7f6ce2697b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank25]: frame #20: + 0xe1df28 (0x7f6cf2183f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank25]: frame #21: + 0x4cf144 (0x7f6cf1835144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank25]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank25]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank25]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank25]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank25]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank25]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank25]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank25]: frame #41: + 0x29d90 (0x7f6d08babd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank25]: frame #42: __libc_start_main + 0x80 (0x7f6d08babe40 in /lib/x86_64-linux-gnu/libc.so.6) [rank25]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank25]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank11]:[W1114 14:39:04.483590510 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 11] using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. srun: error: h100-ord01-03-1023: task 25: Exited with exit code 1 [rank11]: Traceback (most recent call last): [rank11]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank11]: main() [rank11]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank11]: dist.barrier() [rank11]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank11]: return func(*args, **kwargs) [rank11]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank11]: work = group.barrier(opts=opts) [rank11]: torch.distributed.DistBackendError: [11] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank11]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank11]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f834b16bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank11]: frame #1: + 0x6021e58 (0x7f833ace1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f833acdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f833acdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f833acdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f833ac8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f833ac8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f833ac8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f82ed95adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank11]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f82ed9674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank11]: frame #10: + 0x1205255 (0x7f82ed96d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank11]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f82ed96e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank11]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f82ed97f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank11]: frame #13: + 0x5fc28a2 (0x7f833ac828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #14: + 0x5fcd035 (0x7f833ac8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #15: + 0x55d8ceb (0x7f833a298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #16: + 0x55d6579 (0x7f833a296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #17: + 0x1aa7828 (0x7f8336767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #18: + 0x5fd6d74 (0x7f833ac96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #19: + 0x5fd7b05 (0x7f833ac97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank11]: frame #20: + 0xe1df28 (0x7f834a783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank11]: frame #21: + 0x4cf144 (0x7f8349e35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank11]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank11]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank11]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank11]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank11]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank11]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank11]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank11]: frame #41: + 0x29d90 (0x7f8361188d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank11]: frame #42: __libc_start_main + 0x80 (0x7f8361188e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank11]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank11]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-629: task 11: Exited with exit code 1 [rank45]:[W1114 14:39:08.895650057 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 45] using GPU 5 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank45]: Traceback (most recent call last): [rank45]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank45]: main() [rank45]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank45]: dist.barrier() [rank45]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank45]: return func(*args, **kwargs) [rank45]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank45]: work = group.barrier(opts=opts) [rank45]: torch.distributed.DistBackendError: [45] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank45]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank45]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f1cc5cb8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank45]: frame #1: + 0x6021e58 (0x7f1cb58e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f1cb58debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f1cb58def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f1cb58dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1cb588ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1cb588ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1cb588ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f1c6855adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank45]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f1c685674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank45]: frame #10: + 0x1205255 (0x7f1c6856d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank45]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f1c6856e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank45]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f1c6857f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank45]: frame #13: + 0x5fc28a2 (0x7f1cb58828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #14: + 0x5fcd035 (0x7f1cb588d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #15: + 0x55d8ceb (0x7f1cb4e98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #16: + 0x55d6579 (0x7f1cb4e96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #17: + 0x1aa7828 (0x7f1cb1367828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #18: + 0x5fd6d74 (0x7f1cb5896d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #19: + 0x5fd7b05 (0x7f1cb5897b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank45]: frame #20: + 0xe1df28 (0x7f1cc5383f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank45]: frame #21: + 0x4cf144 (0x7f1cc4a35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank45]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank45]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank45]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank45]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank45]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank45]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank45]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank45]: frame #41: + 0x29d90 (0x7f1cdbcf6d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank45]: frame #42: __libc_start_main + 0x80 (0x7f1cdbcf6e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank45]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank45]: . This may indicate a possible application crash on rank 0 or a network set up issue. [rank63]:[W1114 14:39:08.210584783 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 63] using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank63]: Traceback (most recent call last): [rank63]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank63]: main() [rank63]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank63]: dist.barrier() [rank63]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank63]: return func(*args, **kwargs) [rank63]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank63]: work = group.barrier(opts=opts) [rank63]: torch.distributed.DistBackendError: [63] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank63]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank63]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f691e36bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank63]: frame #1: + 0x6021e58 (0x7f690dee1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f690dedebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f690dedef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f690dedfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f690de8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f690de8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f690de8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f68c0b5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank63]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f68c0b674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank63]: frame #10: + 0x1205255 (0x7f68c0b6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank63]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f68c0b6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank63]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f68c0b7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank63]: frame #13: + 0x5fc28a2 (0x7f690de828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #14: + 0x5fcd035 (0x7f690de8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #15: + 0x55d8ceb (0x7f690d498ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #16: + 0x55d6579 (0x7f690d496579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #17: + 0x1aa7828 (0x7f6909967828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #18: + 0x5fd6d74 (0x7f690de96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #19: + 0x5fd7b05 (0x7f690de97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank63]: frame #20: + 0xe1df28 (0x7f691d983f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank63]: frame #21: + 0x4cf144 (0x7f691d035144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank63]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank63]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank63]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank63]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank63]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank63]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank63]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank63]: frame #41: + 0x29d90 (0x7f69343c3d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank63]: frame #42: __libc_start_main + 0x80 (0x7f69343c3e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank63]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank63]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-1017: task 45: Exited with exit code 1 srun: error: h100-ord01-03-948: task 63: Exited with exit code 1 [rank3]:[W1114 14:39:15.842136048 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 3] using GPU 3 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank3]: Traceback (most recent call last): [rank3]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank3]: main() [rank3]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank3]: dist.barrier() [rank3]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank3]: return func(*args, **kwargs) [rank3]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank3]: work = group.barrier(opts=opts) [rank3]: torch.distributed.DistBackendError: [3] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank3]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank3]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f594a8b8cf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank3]: frame #1: + 0x6021e58 (0x7f593a4e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f593a4debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f593a4def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f593a4dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f593a48ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f593a48ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f593a48ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f58ed15adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank3]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f58ed1674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank3]: frame #10: + 0x1205255 (0x7f58ed16d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank3]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f58ed16e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank3]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f58ed17f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank3]: frame #13: + 0x5fc28a2 (0x7f593a4828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #14: + 0x5fcd035 (0x7f593a48d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #15: + 0x55d8ceb (0x7f5939a98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #16: + 0x55d6579 (0x7f5939a96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #17: + 0x1aa7828 (0x7f5935f67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #18: + 0x5fd6d74 (0x7f593a496d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #19: + 0x5fd7b05 (0x7f593a497b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank3]: frame #20: + 0xe1df28 (0x7f5949f83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank3]: frame #21: + 0x4cf144 (0x7f5949635144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank3]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank3]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank3]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank3]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank3]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank3]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank3]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank3]: frame #41: + 0x29d90 (0x7f5960887d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank3]: frame #42: __libc_start_main + 0x80 (0x7f5960887e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank3]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank3]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-801: task 3: Exited with exit code 1 [rank15]:[W1114 14:39:33.487714603 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 15] using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank15]: Traceback (most recent call last): [rank15]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank15]: main() [rank15]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank15]: dist.barrier() [rank15]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank15]: return func(*args, **kwargs) [rank15]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank15]: work = group.barrier(opts=opts) [rank15]: torch.distributed.DistBackendError: [15] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank15]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank15]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd37f16bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank15]: frame #1: + 0x6021e58 (0x7fd36ece1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7fd36ecdebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7fd36ecdef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7fd36ecdfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd36ec8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd36ec8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7fd36ec8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7fd32195adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank15]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7fd3219674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank15]: frame #10: + 0x1205255 (0x7fd32196d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank15]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7fd32196e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank15]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7fd32197f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank15]: frame #13: + 0x5fc28a2 (0x7fd36ec828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #14: + 0x5fcd035 (0x7fd36ec8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #15: + 0x55d8ceb (0x7fd36e298ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #16: + 0x55d6579 (0x7fd36e296579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #17: + 0x1aa7828 (0x7fd36a767828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #18: + 0x5fd6d74 (0x7fd36ec96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #19: + 0x5fd7b05 (0x7fd36ec97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank15]: frame #20: + 0xe1df28 (0x7fd37e783f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank15]: frame #21: + 0x4cf144 (0x7fd37de35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank15]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank15]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank15]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank15]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank15]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank15]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank15]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank15]: frame #41: + 0x29d90 (0x7fd3951f8d90 in /lib/x86_64-linux-gnu/libc.so.6) [rank15]: frame #42: __libc_start_main + 0x80 (0x7fd3951f8e40 in /lib/x86_64-linux-gnu/libc.so.6) [rank15]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank15]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-629: task 15: Exited with exit code 1 [rank57]:[W1114 14:40:23.707652233 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 57] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank57]: Traceback (most recent call last): [rank57]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank57]: main() [rank57]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank57]: dist.barrier() [rank57]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank57]: return func(*args, **kwargs) [rank57]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank57]: work = group.barrier(opts=opts) [rank57]: torch.distributed.DistBackendError: [57] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank57]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank57]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f7c26f6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank57]: frame #1: + 0x6021e58 (0x7f7c16ae1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f7c16adebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f7c16adef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f7c16adfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7c16a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7c16a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f7c16a8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f7bc975adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank57]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f7bc97674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank57]: frame #10: + 0x1205255 (0x7f7bc976d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank57]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f7bc976e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank57]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f7bc977f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank57]: frame #13: + 0x5fc28a2 (0x7f7c16a828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #14: + 0x5fcd035 (0x7f7c16a8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #15: + 0x55d8ceb (0x7f7c16098ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #16: + 0x55d6579 (0x7f7c16096579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #17: + 0x1aa7828 (0x7f7c12567828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #18: + 0x5fd6d74 (0x7f7c16a96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #19: + 0x5fd7b05 (0x7f7c16a97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank57]: frame #20: + 0xe1df28 (0x7f7c26583f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank57]: frame #21: + 0x4cf144 (0x7f7c25c35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank57]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank57]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank57]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank57]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank57]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank57]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank57]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank57]: frame #41: + 0x29d90 (0x7f7c3cf9dd90 in /lib/x86_64-linux-gnu/libc.so.6) [rank57]: frame #42: __libc_start_main + 0x80 (0x7f7c3cf9de40 in /lib/x86_64-linux-gnu/libc.so.6) [rank57]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank57]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-948: task 57: Exited with exit code 1 [rank7]:[W1114 14:40:55.307387117 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 7] using GPU 7 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank7]: Traceback (most recent call last): [rank7]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank7]: main() [rank7]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank7]: dist.barrier() [rank7]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank7]: return func(*args, **kwargs) [rank7]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank7]: work = group.barrier(opts=opts) [rank7]: torch.distributed.DistBackendError: [7] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank7]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank7]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f6ae3d6bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank7]: frame #1: + 0x6021e58 (0x7f6ad38e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f6ad38debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f6ad38def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f6ad38dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6ad388ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6ad388ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f6ad388ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f6a8655adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank7]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f6a865674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank7]: frame #10: + 0x1205255 (0x7f6a8656d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank7]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f6a8656e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank7]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f6a8657f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank7]: frame #13: + 0x5fc28a2 (0x7f6ad38828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #14: + 0x5fcd035 (0x7f6ad388d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #15: + 0x55d8ceb (0x7f6ad2e98ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #16: + 0x55d6579 (0x7f6ad2e96579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #17: + 0x1aa7828 (0x7f6acf367828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #18: + 0x5fd6d74 (0x7f6ad3896d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #19: + 0x5fd7b05 (0x7f6ad3897b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank7]: frame #20: + 0xe1df28 (0x7f6ae3383f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank7]: frame #21: + 0x4cf144 (0x7f6ae2a35144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank7]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank7]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank7]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank7]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank7]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank7]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank7]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank7]: frame #41: + 0x29d90 (0x7f6af9dfed90 in /lib/x86_64-linux-gnu/libc.so.6) [rank7]: frame #42: __libc_start_main + 0x80 (0x7f6af9dfee40 in /lib/x86_64-linux-gnu/libc.so.6) [rank7]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank7]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-801: task 7: Exited with exit code 1 [rank9]:[W1114 14:42:38.768489383 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 9] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank9]: Traceback (most recent call last): [rank9]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank9]: main() [rank9]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank9]: dist.barrier() [rank9]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank9]: return func(*args, **kwargs) [rank9]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank9]: work = group.barrier(opts=opts) [rank9]: torch.distributed.DistBackendError: [9] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank9]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank9]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f14a536bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank9]: frame #1: + 0x6021e58 (0x7f1494ee1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f1494edebdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f1494edef5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f1494edfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1494e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1494e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f1494e8ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f1447b5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank9]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f1447b674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank9]: frame #10: + 0x1205255 (0x7f1447b6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank9]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f1447b6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank9]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f1447b7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank9]: frame #13: + 0x5fc28a2 (0x7f1494e828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #14: + 0x5fcd035 (0x7f1494e8d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #15: + 0x55d8ceb (0x7f1494498ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #16: + 0x55d6579 (0x7f1494496579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #17: + 0x1aa7828 (0x7f1490967828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #18: + 0x5fd6d74 (0x7f1494e96d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #19: + 0x5fd7b05 (0x7f1494e97b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank9]: frame #20: + 0xe1df28 (0x7f14a4983f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank9]: frame #21: + 0x4cf144 (0x7f14a4035144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank9]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank9]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank9]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank9]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank9]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank9]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank9]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank9]: frame #41: + 0x29d90 (0x7f14bb36ad90 in /lib/x86_64-linux-gnu/libc.so.6) [rank9]: frame #42: __libc_start_main + 0x80 (0x7f14bb36ae40 in /lib/x86_64-linux-gnu/libc.so.6) [rank9]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank9]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-629: task 9: Exited with exit code 1 [rank41]:[W1114 14:42:47.130347921 ProcessGroupNCCL.cpp:4268] [PG ID 0 PG GUID 0 Rank 41] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. Specify device_ids in barrier() to force use of a particular device, or call init_process_group() with a device_id. [rank41]: Traceback (most recent call last): [rank41]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 552, in [rank41]: main() [rank41]: File "/home/tony/Work/tony/RealGen/encode_semantic_codes.py", line 513, in main [rank41]: dist.barrier() [rank41]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper [rank41]: return func(*args, **kwargs) [rank41]: File "/home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 4376, in barrier [rank41]: work = group.barrier(opts=opts) [rank41]: torch.distributed.DistBackendError: [41] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: failed to recv, got 0 bytes [rank41]: Exception raised from recvBytes at ../torch/csrc/distributed/c10d/Utils.hpp:670 (most recent call first): [rank41]: frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f415156bcf6 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libc10.so) [rank41]: frame #1: + 0x6021e58 (0x7f41410e1e58 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #2: c10d::TCPStore::doWait(c10::ArrayRef, std::chrono::duration >) + 0x35b (0x7f41410debdb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #3: c10d::TCPStore::doGet(std::string const&) + 0x2a (0x7f41410def5a in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #4: c10d::TCPStore::get(std::string const&) + 0x7a (0x7f41410dfcfa in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #5: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f414108ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #6: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f414108ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #7: c10d::PrefixStore::get(std::string const&) + 0x31 (0x7f414108ffb1 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #8: c10d::ProcessGroupNCCL::broadcastUniqueNCCLID(ncclUniqueId*, bool, std::string const&, int) + 0xaf (0x7f40f3d5adef in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank41]: frame #9: c10d::ProcessGroupNCCL::getNCCLComm(std::string const&, c10::Device&, c10d::OpType, int, bool) + 0x10e5 (0x7f40f3d674f5 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank41]: frame #10: + 0x1205255 (0x7f40f3d6d255 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank41]: frame #11: c10d::ProcessGroupNCCL::allreduce_impl(at::Tensor&, c10d::AllreduceOptions const&) + 0x12c (0x7f40f3d6e88c in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank41]: frame #12: c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&) + 0x45e (0x7f40f3d7f91e in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so) [rank41]: frame #13: + 0x5fc28a2 (0x7f41410828a2 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #14: + 0x5fcd035 (0x7f414108d035 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #15: + 0x55d8ceb (0x7f4140698ceb in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #16: + 0x55d6579 (0x7f4140696579 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #17: + 0x1aa7828 (0x7f413cb67828 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #18: + 0x5fd6d74 (0x7f4141096d74 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #19: + 0x5fd7b05 (0x7f4141097b05 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so) [rank41]: frame #20: + 0xe1df28 (0x7f4150b83f28 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank41]: frame #21: + 0x4cf144 (0x7f4150235144 in /home/tony/anaconda3/envs/suno_env_dev/lib/python3.10/site-packages/torch/lib/libtorch_python.so) [rank41]: frame #22: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4fcaf7] [rank41]: frame #23: _PyObject_MakeTpCall + 0x25b (0x4f657b in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #24: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x50861f] [rank41]: frame #25: _PyEval_EvalFrameDefault + 0x13b2 (0x4ee722 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #26: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #27: _PyEval_EvalFrameDefault + 0x2de4 (0x4f0154 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #28: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #29: _PyEval_EvalFrameDefault + 0x4b2c (0x4f1e9c in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #30: _PyFunction_Vectorcall + 0x6f (0x4fcf3f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #31: _PyEval_EvalFrameDefault + 0x31f (0x4ed68f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #32: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5924f2] [rank41]: frame #33: PyEval_EvalCode + 0x87 (0x592437 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #34: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5c3237] [rank41]: frame #35: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5be380] [rank41]: frame #36: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x4598d6] [rank41]: frame #37: _PyRun_SimpleFileObject + 0x19f (0x5b890f in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #38: _PyRun_AnyFileObject + 0x43 (0x5b8673 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #39: Py_RunMain + 0x38d (0x5b542d in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #40: Py_BytesMain + 0x39 (0x585609 in /home/tony/anaconda3/envs/suno_env_dev/bin/python3) [rank41]: frame #41: + 0x29d90 (0x7f416752ad90 in /lib/x86_64-linux-gnu/libc.so.6) [rank41]: frame #42: __libc_start_main + 0x80 (0x7f416752ae40 in /lib/x86_64-linux-gnu/libc.so.6) [rank41]: frame #43: /home/tony/anaconda3/envs/suno_env_dev/bin/python3() [0x5854be] [rank41]: . This may indicate a possible application crash on rank 0 or a network set up issue. srun: error: h100-ord01-03-1017: task 41: Exited with exit code 1