TL;DR Based on long context KLD benchmarks, KVarN appears to be just better than usual llama.cpp KV cache quants. At every size, KVarN matches precision of usual quants of one bit higher. A number of people in the comments under my previous post asked a fair question: what if we drop the obsession with 2-bit and 3-bit toy quants and apply KVarN to high end? So I did just that in my latest BeeLlama v0.3.2 Preview (fork of llama.cpp with DFlash, in short) and ran the same benchmarks a...