
Tech
Why Local LLMs Suddenly Slow Down at Long Context
Your local LLM runs fine until the context fills up past a certain point - then generation speed can drop by ~50%. The cause is the KV cache spilling out of VRAM into slower shared memory. On Windows it happens silently, with no out-of-memory error to warn you.
Read the full discussion on HackerNoon
This article was aggregated from HackerNoon. Click to join the conversation.
View on HackerNoon