performance-gradient-accumulatio-training-loss-becomes-unusually-large-when-gradien-78b50040

THEN Ensure the loss is correctly divided by the number of gradient accumulation steps, applied only once before backward. Review custom loss functions that inherit from base classes to avoid multiplying scaling factors due to typographical errors.

# Correct loss scaling for gradient accumulation (PyTorch)
loss = loss / accumulation_steps
loss.backward()
if (step + 1) % accumulation_steps == 0:
    optimizer.step()
    optimizer.zero_grad()