Tech
[Local LLM] RTX PRO 5000(或其他 48GB 显存)的 Qwen3.8-27B-FP8 配置交流(prefill 5000+t/s, decode 60+t/s)
先上参数:
python -m sglang.launch_server \
--model-path Qwen3.8-27B-FP8 \
--attention-backend flashinfer \
--kv-cache-dtype fp8_e4m3 \
--mamba-radix-cache-strategy extra_buffer_lazy \
--mamba-full-memory-ratio 1.0 \
--chunked-prefill-size 2048 \
--context-length 262144 \
--mem-fraction-static 0.90 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder \
--mm-feature-transport cpu \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-top...
Read the full discussion on V2EX
This article was aggregated from V2EX. Click to join the conversation.
View on V2EX