We don't publish
your competitive advantage.

IFWhen using OLMoE or GPT-Oss models with top_k > 1, the auxiliary load‑balancing loss is computed incorrectly: the fraction of tokens routed per expert (f_i) is not divided by top_k, causing the loss to be too high by a factor of top_k.

THENModify the auxiliary loss computation to normalize f_i by top_k (K). In the snippet where tokens_per_expert is computed, divide the sum by K: tokens_per_expert = torch.sum(expert_mask.float() * expert_attention_mask, dim=0) / (torch.sum(expert_attention_mask, dim=0) * K). This ensures both the load distribution f_i and the softmax probability P_i have the same scale, fixing the imbalance signal.

IFAuxiliary balancing loss in MoE models like OLMoE and GPT-Oss is incorrectly computed without dividing by top_k, causing the loss to be too high by a factor of K.

THENNormalize the fraction of tokens routed to each expert (f_i) by dividing by top_k (K). This ensures f_i and P_i represent the same distribution, correcting the auxiliary loss magnitude. Update the code so that f_i = (N / (T*K)) * sum(1{expert chosen}), rather than using N/T only.