← sources

Two days tuning a single Tesla P40: 35B MoE, full 262k context, ~60 t/s short and ~30 t/s at 100k deep. No sub-Q4 quants.

source: reddit (reddit:LocalLLaMA)published 2026-10-28 10:58ingested 2026-10-29 02:42

retained text

i Got a bit obsessed with my P40 this week trying to max speed on Ornith 1.5 35B-A3B (qwen3.5 MoE arch) without going below Q4 weights or q8 KV. Shrinking quants to win a benchmark has not really been useful to me so ai wanted a logical middle ground that was still “fast”

Setup: one Tesla P40 24GB on PCIe x1 (yes, x1), Windows, my own llama.cpp fork. Q4\_0 built from the official BF16, MTP draft (n=2), full 262k context, q8\_0 KV.

What actually moved it:

\- **ECC off +**1.5 GB VRAM and \~9% decode.  
\- *q8\_0 KV, not q4\_0.* On Pascal q4 KV falls apart with depth (30.6 vs 42.2 t/s at 32k).   
\- **Check your MTP head.** The first upload of this model, and quants built from it, have an untrained MTP head. \`blk.40.nextn.shared\_head\_norm\` mean \~1.02 = untrained, \~2.93 = trained. Draft acceptance went from \~45% to 60-85%.  
\- **int8 dp4a for the KQ half of the flash-attention tile kernel.** Pascal has no tensor cores and crippled fp16, but dp4a is fast. Prefill +26% at 32k, +41% at 100k.  
\- **MTP prefill window.** The MTP head is a full-attention layer, so on long prompts it costs \~13% prefill. Letting it read only the last 8k of the prompt gets that back (121k prompt: 327 -> 373 t/s). Output doesn't change, the main model still verifies every draft.

Didn't work: f16 KV (faster in llama-bench, but slower and less context once MTP is on in a real server), dequantizing V inside the kernel (-5%), the official Q4\_K\_M (same quality score as my Q4\_0, slower, less context).

llama-bench, no MTP, 2 runs each:

| depth | prefill t/s | decode t/s |  
|---|---|---|  
| 0 | 1104 | 60.8 |  
| 8k | 880 | 57.0 |  
| 32k | 552 | 46.0 |  
| 64k | 366 | 35.8 |  
| 100k | 265 | 29.1 |  
| 190k | - | 18.8 |

Real server with MTP: 65-74 t/s on short prompts. A 101k-token prompt reads at \~415 t/s and generates at 35-39 t/s.

I couldn't find anyone else posting P40 numbers at depth, so hopefully the curve helps someone. Happy to share the patches if you want to try them on your own Pascal card.

document fetched 2026-10-29 00:44 · canonical https://reddit.com/r/LocalLLaMA/comments/1xhkwlm/two_days_tuning_a_single_tesla_p40_35b_moe_full