ai-agents-model-loading-when-loading-a-fine-tuned-model-that-includes-a-bn-8676dc3d

THEN Remove the BNB quantization configuration from the model's config before loading with vLLM. In the model's config.json, set the 'quantization_config' field to null or delete the key. Alternatively, load the model without the quantization config using transformers and then pass the resulting model to vLLM.

import json
# Load the config
with open('config.json') as f:
    config = json.load(f)
# Remove quantization config
config.pop('quantization_config', None)
# Save back
with open('config.json', 'w') as f:
    json.dump(config, f, indent=2)