# Running 35B MoE models on a Tesla K80 (2014) at ~21 tok/s, in plain C — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/5ea3181d4e5fcca1

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-10-20 17:49 ingested 2026-10-21 02:41

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1xaz06x/running_35b_moe_models_on_a_tesla_k80_2014_at_21/)

## retained text



```
I've spent about a year on an inference engine that doesn't need modern hardware. It's called **Pantheon** (LLM engine) on top of **Foundry** (ML library), both in C with no outside libraries. The CUDA backend targets Kepler (`sm_37`), so a Tesla K80 you can buy used for very little does real work.

**Measured on one K80 (both dies) in a 2-core, 8 GB, no-AVX host:**

* Qwen3.6-35B-A3B and Nex-N2-mini (Q3\_K\_M): **\~21 tok/s** decode
* Prompt reading: **\~110-120 tok/s** up to \~8k tokens; \~75 at 14k, \~55 at 30k
* Weights live on the card, not in system RAM, which is why an 8 GB host is enough

**Prefix caching is built in.** Send a session id and later turns only process what's new. opencode does this on its own: its second turn (14k-token system prompt) went from 170 s to 9 s.

**Honest limits:** prefill is the K80's weak spot. Chat, writing, Q&A and background jobs feel fine; pasting a 30k-token codebase every request means minutes before the first word.

    git clone --recursive https://git.teamide.dev/David/Pantheon.git
    cd Pantheon && ./install.sh --k80
    ./k80_qwen36.sh      # or ./k80_nexn2.sh

[`install.sh`](http://install.sh) `--k80` builds it (no CUDA toolkit needed, just the NVIDIA 470 driver). The run script downloads and checks the model, then serves an OpenAI-compatible API.

🔗 Pantheon: [https://git.teamide.dev/David/Pantheon](https://git.teamide.dev/David/Pantheon)

🔗 Foundry: [https://git.teamide.dev/David/Foundry](https://git.teamide.dev/David/Foundry)

🤗 [https://huggingface.co/dcherrera/Qwen3.6-35B-A3B-Pantheon-K80](https://huggingface.co/dcherrera/Qwen3.6-35B-A3B-Pantheon-K80) [https://huggingface.co/dcherrera/Nex-N2-mini-Pantheon-K80](https://huggingface.co/dcherrera/Nex-N2-mini-Pantheon-K80)

It's early and I want feedback: build problems, bugs, anything confusing. Open an issue or PR on [git.teamide.dev](http://git.teamide.dev) (sign in with GitHub), or DM me. It also runs Ornith 1.5 on Tenstorrent Blackhole cards, if you have one of those.
```



document fetched 2026-10-21 00:41 · canonical https://reddit.com/r/LocalLLaMA/comments/1xaz06x/running_35b_moe_models_on_a_tesla_k80_2014_at_21
