infrastructure-deepspeed-zero3-pret-when-loading-a-pretrained-model-with-deepspeed-zer-aa36b311

THEN For DeepSpeed ZeRO-3, avoid using `ignore_mismatched_sizes=True` directly. Instead, load the model without that flag (which triggers automatic weight consolidation), then manually adjust the classifier head (e.g., replace the output layer with the desired number of classes) after loading, or use `deepspeed.zero.GatheredParameters` to gather sharded weights before modification.

# Instead of:
# model = AutoModelForSequenceClassification.from_pretrained(model_name, ignore_mismatched_sizes=True)
# Do:
model = AutoModelForSequenceClassification.from_pretrained(model_name)
# Then replace the classifier head manually if output size differs:
model.classifier = torch.nn.Linear(model.config.hidden_size, num_labels)