infrastructure-distributed-training-when-using-deepspeed-with-transformers-trainer-the-5dae7a15

THEN Set the environment variable NCCL_TIMEOUT to the desired timeout in milliseconds before launching the training script (e.g., NCCL_TIMEOUT=3600000). Alternatively, manually initialize the distributed process group with the correct timeout using torch.distributed.init_process_group(..., timeout=timedelta(seconds=3600)) before constructing the Trainer.

import os
os.environ['NCCL_TIMEOUT'] = '3600000'  # milliseconds