infrastructure-distributed-deadlock-a-ray-worker-raises-an-exception-e-g-torch-cuda-se-6a76daf7

THEN To prevent deadlock, use multithreading: one thread to call ray.wait() to monitor worker exceptions, and another thread to call dist.init_process_group(). Alternatively, design initialization to handle exceptions before any distributed synchronization point—ensure all workers successfully complete initialization tasks (e.g., CUDA device setting) before entering process group creation.

import concurrent.futures

def init_worker(rank, world_size, init_method):
    # This may raise exception
    torch.cuda.set_device(rank)
    dist.init_process_group(backend='gloo', init_method=init_method, world_size=world_size, rank=rank)

def check_worker_exception(handle):
    try:
        ray.get(handle)
    except Exception as e:
        print(f'Worker failed: {e}')
        # terminate main process group init
        exit(1)

with concurrent.futures.ThreadPoolExecutor() as executor:
    worker_future = executor.submit(check_worker_exception, handle)
    main_future = executor.submit(dist.init_process_group, backend='gloo', init_method=init_method, world_size=world_size, rank=1)
    # Complete both or handle first exception