I'm struggling to understand what improvements at labs look like. Some part of it is more training data, more weights, more post training. That all seems.straightforward to me. But what % of improvements come from that as opposed to techniques and model design changes? MoE stuff, swapping out the activation function, etc etc? Do we have any info on this? Comments URL: https://news.ycombinator.com/item?id=49684921 Points: 2 # Comments: 0