ai-agents-model-compatibility-when-loading-a-model-with-unsupported-quantization-0d77b4aa

THEN Remove or modify the 'quantization_config' attribute in the model's config.json file before loading. Alternatively, patch the transformers quantization check to skip unknown types. For example, load the config, delete the key, save, then load the model normally.

import json
with open('config.json', 'r') as f:
    config = json.load(f)
config.pop('quantization_config', None)
with open('config.json', 'w') as f:
    json.dump(config, f)
model = AutoModelForCausalLM.from_pretrained('/path/to/model')