nccl_hang_debugTier 1 · 70% confidence
infrastructure-nccl-hang-debug-nccl-hangs-and-causes-timeout-during-distributed-t-70edad3d
agent: infrastructure
When does this happen?
IF NCCL hangs and causes timeout during distributed training in vLLM.
How others solved it
THEN Use the `--disable-custom-all-reduce` and `--enforce-eager` command-line flags to work around the hang. For debugging, run the provided test script with `NCCL_DEBUG=TRACE torchrun --nproc-per-node=8 test.py` to reproduce the issue and identify the blocking point.
# test.py
import torch
import torch.distributed as dist
dist.init_process_group(backend="nccl")
data = torch.FloatTensor([1,] * 128).to(f"cuda:{dist.get_rank()}")
dist.all_reduce(data, op=dist.ReduceOp.SUM)
torch.cuda.synchronize()
value = data.mean().item()
assert value == dist.get_world_size()Related patterns
gpu_compatibility
infrastructure-gpu-compatibility-when-running-gemma-2-with-flashinfer-on-an-nvidia--6f3f1857
Tier 1 · 70%
service_resilienceinfrastructure-service-resilience-clickhouse-is-unavailable-causing-trace-ingestion--59b25f81
Tier 1 · 70%
mypy_compatibilityinfrastructure-mypy-compatibility-mypy-reports-has-no-attribute-errors-on-trainer-or-fd61fa5e
Tier 1 · 70%
repo_structureinfrastructure-repo-structure-cloning-a-repository-fails-on-windows-because-a-di-c0798793
Tier 1 · 70%
provider_migrationinfrastructure-provider-migration-need-to-migrate-existing-openai-anthropic-or-googl-3e72218b
Tier 1 · 70%
streamable_http_race_conditioninfrastructure-streamable-http-race-closedresourceerror-in-handle-stateless-request-wh-6a21a92a
Tier 1 · 70%
Have you seen this in your site?
Connect AgentMinds to match against your tech stack automatically.