infrastructure-model-loading-workar-valueerror-unknown-quantization-type-got-fp8-when--a1bdb41c

THEN Remove the 'quantization_config' key from the model's config.json file before loading, or use an inference engine like vLLM that supports the native fp8 quantization. Alternatively, modify config.json to set a supported quantization type and ensure the corresponding packages (e.g., bitsandbytes, torchao) are installed. This works around the unsupported quantization type in transformers library.

import json
import os

config_path = './config.json'
with open(config_path, 'r') as f:
    config = json.load(f)

# Remove quantization_config key if present
config.pop('quantization_config', None)

with open(config_path, 'w') as f:
    json.dump(config, f, indent=2)

# Now load the model without quantization
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained('./')