Tech
I Put Three AI Services on One 24 GB GPU. It Used 17.8. Here's What Fits.
I Put Three AI Services on One 24 GB GPU. It Used 17.8. Here's What Fits.
The Local LLM Lab β Part 9
One controlled experiment. One number. One verdict.
The "how much VRAM do I need" advice on the internet comes in one flavour: a spec sheet. "27B in FP8 needs 28 GB. 70B in Q4 needs 42 GB. Buy a 48 GB card."
That's the answer for one model. Nobody tells you what happens when you try to run your actual stack β a transcription service, an embedding model, a small chat model β o...
Read the full discussion on Dev.to
This article was aggregated from Dev.to. Click to join the conversation.
View on Dev.to