Introducing ReMoE: a MoE architecture with fully differentiable ReLU router. It consistently outperforms TopK across various model sizes and granularities, with stronger gains as the number of experts grows.
Paper: arxiv.org/abs/2412.14711
Github: github.com/thu-ml/ReMoE