LocalLLaMA
New sampler + verifier *drastically* improves tiny 0.5b model coding performance
I read it with a little bit of effort The tiny model result is insane, theoretically this could make make a 0.5b on-par with a 2/3/4b ish class model in coding with no weights change*. And for large models it could maybe fix let's say 30-50% hallucination problems (educated guesstimate here) Don't expect this to ever come to vLLM or SGLang, but llama.cpp could integrate this easily* like `--top-n-sigma`. *Now there's this one... small... okay big catch: Aside from this being a backtrack...
Read the full discussion on Reddit
This article was aggregated from Reddit. Click to join the conversation.
View on Reddit