ai-agents-auxiliary-loss-norma-when-using-olmoe-or-gpt-oss-models-with-top-k-1-th-16fed64f

THEN Modify the auxiliary loss computation to normalize f_i by top_k (K). In the snippet where tokens_per_expert is computed, divide the sum by K: tokens_per_expert = torch.sum(expert_mask.float() * expert_attention_mask, dim=0) / (torch.sum(expert_attention_mask, dim=0) * K). This ensures both the load distribution f_i and the softmax probability P_i have the same scale, fixing the imbalance signal.

# Corrected normalization for top_k > 1
K = config.num_experts_per_tok  # example: top_k
expert_attention_mask = ...
tokens_per_expert = torch.sum(expert_mask.float() * expert_attention_mask, dim=0) / (torch.sum(expert_attention_mask, dim=0) * K)