LocalLLaMA
ggml-webgpu: Improve prefill speeds for k-quants + refactor matmul for Q4/Q5/Q8 and k-quants by yomaytk · Pull Request #24225 · ggml-org/llama.cpp
This PR improves matmul performance for k-quants. The following table shows the improvement on the pp512 test in M2 pro. quant model master (t/s) PR (t/s) speedup Q2_K qwen3 0.6B Q2_K - Medium 817.86 ± 6.14 1991.81 ± 6.87 2.44x Q3_K qwen35 4B Q3_K - Medium 92.54 ± 0.13 302.24 ± 0.37 3.27x gemma4 E4B Q3_K - Medium 79.06 ± 0.08 298.73 ± 0.90 3.78x Q4_K qwen35 4B Q4_K - Medium 243.82 ± 0.09 327.24 ± 0.59 1.34x g...
Read the full discussion on Reddit
This article was aggregated from Reddit. Click to join the conversation.
View on Reddit
Related Trends in LocalLLaMA
Building an open-source control plane for self-hosted vLLM, what would you want in it?
5090: Windows or Linux for Qwen3.8.27b
RTX Pro 4500 vs. A100 (x4) - purchase being made today
Best llm at understaning middle eastern languages? Arabic/Persian etc
Qwen3.8-27B abliterated FP8: refusal 64–99% → 0–6%, and MMLU/GSM8K move less than 1.3 points
llama.cpp Windows Manager