LocalLLaMA
Build for local LLM with 2 separate GPUs
I want to build a headless compute machine to run a RTX Ada 4000 (20GB) with a RTX Pro 5000 (48GB) or RTX PRO 4500 (32GB) in parallel for inference. The goal is not running one large model using 2x GPUs, but rather running separate models on each GPU. Why these GPU config? because I already had a RTX Ada 4000 and don't want to sell it for now, but it's not enough to run larger models. This is going to be 95% time for inference and 5% occasional fine tuning LoRA / QLoRA type. This machine ...
Read the full discussion on Reddit
This article was aggregated from Reddit. Click to join the conversation.
View on Reddit