LocalLLaMA
Can you explain the concept behind each of the main size ranges of LLM models, as in, what hardware setups the different size niches are meant to fit into (~30b, ~70b, ~120b, ~230b, etc). Like is it mainly based on pro hardware sizing for 8-bit, or consumer GPU vram for ~Q4, or some mixture?
I am curious about intended sizings of the main size niches of the popular local LLM models. As in, we can see there is a major niche at 26b-35b, then hardly anything from 36 through 69b, then (formerly) another major niche at ~70b-72b, then another niche at ~120b-123b, then another big gap till ~230b-235b, and then it gets a bit more mixed all over the place after that with 300b-750b being scattered more randomly probably based more on just whatever the best strength per size they could get w...
Read the full discussion on Reddit
This article was aggregated from Reddit. Click to join the conversation.
View on Reddit