On the CPU, batch 1 is memory bandwidth bound. But if token/s = bandwidth / (bytes_per_weight * active_weights_per_token) the total number of parameters doesnt slow down the generation speed. So building the architecture aroud a small batch "active parameters per token" (ternary weights and a granular MoE), the total capacity can grow without effecting the speed. Now the new catch: if speed is not the problem (a 105M and a 206M model run at the same tok/s (predicted 739–1309 tok/s)),...