infrastructure-distributed-gpu-allo-when-running-quantized-models-e-g-awq-in-a-distrib-161fdbaa

THEN Set the CUDA_VISIBLE_DEVICES environment variable explicitly before initializing the vLLM engine, or upgrade to a version where the quantization code does not override environment variables. As a workaround, manually set CUDA_VISIBLE_DEVICES to the appropriate GPU indices.

import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0,1'  # Set before vLLM engine init