ai-agents-gradient-accumulatio-during-gradient-accumulation-in-lm-training-the-cr-1dc25755

THEN Modify the training loop to compute a list of per-micro-batch item counts (e.g., number of non-ignored labels) and pass the appropriate count to compute_loss for each micro-batch. This ensures consistent gradient magnitude across accumulation steps.

In get_batch_samples, replace scalar num_items_in_batch with a list num_items_in_batches = [(batch['labels'].ne(-100)).sum() for batch in batch_samples]. Then in the training loop, for each micro-batch call loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batches[i]).