ai-agents-quantization-config--when-loading-a-finetuned-model-with-a-bnb-quantiza-43f884d9

THEN Remove or clear the BNB quantization configuration (e.g., the 'quantization_config' key) from the model's configuration file or Hugging Face model card before loading with vLLM. Alternatively, ensure the model is saved without the quantization config if it was used during fine-tuning.

import json
with open('config.json', 'r') as f:
    config = json.load(f)
config.pop('quantization_config', None)
with open('config.json', 'w') as f:
    json.dump(config, f)