
Tech
A 30B Model Crawled on My Mini PC, But the Weights Were Innocent
GLM-4.7-Flash should fly on a small box: 30B total, ~3B active. On a Radeon 780M mini PC it crawled and fell apart past short context. The cause was not the weights or the quant. It was Multi-head Latent Attention with no runtime that implemented it, so the model fell back to plain attention and the KV cache ballooned ~17x. Swapping to ik_llama.cpp with a real MLA path fixed it.
Read the full discussion on HackerNoon
This article was aggregated from HackerNoon. Click to join the conversation.
View on HackerNoon