
LocalLLaMA
Finally finished my LLM server: EPYC 9575F, 4× RTX 3090 (96GB VRAM), 768GB ECC RAM
Took a while, but Nalthis is finally up and assembled. Specs: Supermicro H13SSL-N AMD EPYC 9575F (64C/128T Zen 5) 768GB DDR5-5600 ECC RDIMM 4× RTX 3090 (96GB VRAM total) 1× 2TB NVMe OS 2× 3.94TB NVMe data 2050W ATX 3.1 PSU Corsair 9000D Planned use: vLLM - high throughput small models llamacpp - larger reasoning models I have been making a space simulation and finally ready to integrate AI into how the NPCs doing planning, hoping to get decent throughput on smal...
Read the full discussion on Reddit
This article was aggregated from Reddit. Click to join the conversation.
View on Reddit