LocalLLaMA
What's up on CPU inference these days?
What are the best models, quants and llama.cpp versions/forks for CPU inference these days? I have AVX2 but no AVX512 - Intel core ultra 7 165H; 64G RAM This seems to ask for massive MoE (a lot of RAM, not a lot of bandwidth/compute). So Qwen3.6 35B A3B Q4_K_M with standard llama.cpp produces about 10 tps - usable in non-thinking mode, not usable in thinking mode. Is this the best I can get or are there other options? submitted by /u/ramendik [link] [comme...
Read the full discussion on Reddit
This article was aggregated from Reddit. Click to join the conversation.
View on Reddit