← sources

A year ago I couldn't get vLLM running on my 5090. Now work gave me a DGX Station for a month. What should I run?

source: reddit (reddit:LocalLLaMA)published 2026-11-13 18:27ingested 2026-11-14 02:42

retained text

So about a year ago I was in here asking why vLLM wouldnt run on my 5090 (WSL, flash-infer, the usual pain). These days I'm the AI guy at work, and this week a DGX Station showed up. ASUS ET900N G3, GB300 with \~250 GB VRAM, 72 Grace cores, 750 GB RAM, liquid cooled, plus an RTX PRO 6000 riding along in the same case.

I get it more or less to myself for a month before it becomes the shared inference box for the whole company. Dont want to waste that month.

Some numbers from the first two days, all NVFP4, total tok/s at 1 / 16 / 64 parallel requests

* Qwen3.8-Flash-Next on vLLM 219 / \~1370 / \~2700
* same model on SGLang 221 / 1537 / 3002
* DeepSeek V4.1 Flash on a vLLM nightly 59 single and 246 at 16 paralell (takes 19 minutes to load lol)

The GB300 pulls around 785 W flat out and sits at 50 °C. TensorRT-LLM refuses to install outside the NGC container so thats still on my list. And no, Kimi K3.1 Pro doesnt fit, I checked.

Stuff I was thinking about

1. biggest model that fits and how usable it really is
2. vLLM vs SGLang vs TensorRT-LLM on the same model
3. how many parallel agents it can feed before it falls over

But what do you guys actually want to see on this thing? Long context, video models, finetuning, something dumb? Tell me what to run and how you want it measured and I'll post the results here.

Pics are the inside, the front and my monitoring page (german UI, sorry)

document fetched 2026-11-14 00:46 · canonical https://reddit.com/r/LocalLLaMA/comments/1xvflyc/a_year_ago_i_couldnt_get_vllm_running_on_my_5090