We don't publish
your competitive advantage.

IFWhen gradient accumulation is enabled and training uses a model with loss_kwargs, the loss becomes extremely large due to incorrect scaling of the accumulated loss.

THENEnsure the loss is properly scaled by dividing the accumulated loss by the number of gradient accumulation steps before calling backward. When implementing custom losses with loss_kwargs, verify that the loss tensor is divided by the accumulation steps to prevent large gradients and training instability. Add integration tests with gradient accumulation to catch scaling errors early.