Every LLM inference engineer hits this wall eventually. You deployed a model, it works in testing, then production traffic arrives. Suddenly your 80GB A100 is OOM on a 70B model that "should fit." The culprit is almost always the KV Cache . But most discussions stop at "it caches the Key and Value matrices" — which doesn't help you predict when you'll run out of memory. This post gives you a quick estimator formula, explains when to worry, and what levers actually help. ...