
LocalLLaMA
2X tk/s (from 19.4 -> 38.1 tk/s on 1 x MI50) Playing with a hypothesis like speculative decoding.. but instead of an additional side model, exploiting that I can run multiple computations side-by-side AS IF I had Qwen3.6-27B loaded twice in memory - small quants don't use all the available compute.
MODS: if you wanna remove for slop, that's cool - once I have something like a llama.cpp patch I'll repost something people can use. I'll write a full article on my Medium account with how it works. Just got excited and wanted to share. *** Forgive the claude summary, in the readme, but the base works. I'm still working on the hip kernal and having it combine with MTP. I hope to get up near 80 tk/s. All started because I realized every Q8 (INT8 or F8) calculation was using f32 of compute an...
Read the full discussion on Reddit
This article was aggregated from Reddit. Click to join the conversation.
View on Reddit