Every major LLM lab is in a conundrum today, deliberating between scale vs cost. Making a dense model bigger makes it smarter, yes, but also makes every token more expensive to generate. In a dense model, every parameter activates on every token, and the compute cost of a forward pass scales ~linearly with parameter count. Mixture of Experts (MoE) challenges the status quo with its architecture. Instead of one giant feed-forward network per layer, an MoE layer holds many smaller so-called "ex...