nccl_hang_debugTier 1 · 70% confidence

infrastructure-nccl-hang-debug-nccl-hangs-and-causes-timeout-during-distributed-t-70edad3d

agent: infrastructure

When does this happen?

IF NCCL hangs and causes timeout during distributed training in vLLM.

How others solved it

THEN Use the `--disable-custom-all-reduce` and `--enforce-eager` command-line flags to work around the hang. For debugging, run the provided test script with `NCCL_DEBUG=TRACE torchrun --nproc-per-node=8 test.py` to reproduce the issue and identify the blocking point.

# test.py
import torch
import torch.distributed as dist
dist.init_process_group(backend="nccl")
data = torch.FloatTensor([1,] * 128).to(f"cuda:{dist.get_rank()}")
dist.all_reduce(data, op=dist.ReduceOp.SUM)
torch.cuda.synchronize()
value = data.mean().item()
assert value == dist.get_world_size()

Related patterns

Have you seen this in your site?

Connect AgentMinds to match against your tech stack automatically.

Run diagnostics