← sources

Qwen3.8-27B Jetson Thor: 40 tok/s at FP16-level accuracy (INT4 + MTP + FlashHead)

source: reddit (reddit:LocalLLaMA)published 2026-10-26 17:56ingested 2026-10-27 02:39

retained text

I'm on the team at Embedl that built this, and wanted to share the numbers.

**What's in it**

* INT4 weights, using our own quantization, about half the error of standard AWQ
* FlashHead, our faster LM head: the 262k-vocab output layer is a real cost at batch 1, so it only scores the likely candidates
* MTP head for speculative decoding
* Port to TensorRT Edge-LLM (0.10.1) underneath

**Speed** (batch 1, 2,048-token prompts from real OpenAssistant conversations, 128 out, greedy, average of 20 requests)

* Decode: 39.6 tok/s
* End-to-end: 23.6 tok/s
* Prompt processing: 926 tok/s
* Peak memory: 24.8 GiB

**Accuracy vs the FP16 model** (full test sets, the same engine as served)

* GSM8K: 96.4 (FP16: 96.1)
* IFEval: 83.5 (FP16: 81.9)
* MMLU-Pro: 79.1 (FP16: 79.5)

For comparison: Jetson AI Lab reports about 35 tok/s decode with llama.cpp Q4\_K\_M + MTP. Prompt length isn't stated there, and they don't publish accuracy.  


**To run it**

* embedl infer Qwen/Qwen3.8-27B/thor --prompt "Hello"
* embedl serve Qwen/Qwen3.8-27B/thor (OpenAI-compatible API)

The container download is about 1 GB, and the model weights are a separate download (\~20 GiB).  
  
The full card has power, energy per token and methodology: [https://embedl.com/Qwen/Qwen3.8-27B](https://embedl.com/Qwen/Qwen3.8-27B), **and the ability to test the model on a Thor directly in the browser**  
  
Let us know what model (and hardware) you want to see next!

document fetched 2026-10-27 00:43 · canonical https://reddit.com/r/LocalLLaMA/comments/1xg44nj/qwen3827b_jetson_thor_40_toks_at_fp16level