Qwen3.8-27B Jetson Thor: 40 tok/s at FP16-level accuracy (INT4 + MTP + FlashHead)
retained text
I'm on the team at Embedl that built this, and wanted to share the numbers. **What's in it** * INT4 weights, using our own quantization, about half the error of standard AWQ * FlashHead, our faster LM head: the 262k-vocab output layer is a real cost at batch 1, so it only scores the likely candidates * MTP head for speculative decoding * Port to TensorRT Edge-LLM (0.10.1) underneath **Speed** (batch 1, 2,048-token prompts from real OpenAssistant conversations, 128 out, greedy, average of 20 requests) * Decode: 39.6 tok/s * End-to-end: 23.6 tok/s * Prompt processing: 926 tok/s * Peak memory: 24.8 GiB **Accuracy vs the FP16 model** (full test sets, the same engine as served) * GSM8K: 96.4 (FP16: 96.1) * IFEval: 83.5 (FP16: 81.9) * MMLU-Pro: 79.1 (FP16: 79.5) For comparison: Jetson AI Lab reports about 35 tok/s decode with llama.cpp Q4\_K\_M + MTP. Prompt length isn't stated there, and they don't publish accuracy. **To run it** * embedl infer Qwen/Qwen3.8-27B/thor --prompt "Hello" * embedl serve Qwen/Qwen3.8-27B/thor (OpenAI-compatible API) The container download is about 1 GB, and the model weights are a separate download (\~20 GiB). The full card has power, energy per token and methodology: [https://embedl.com/Qwen/Qwen3.8-27B](https://embedl.com/Qwen/Qwen3.8-27B), **and the ability to test the model on a Thor directly in the browser** Let us know what model (and hardware) you want to see next!
document fetched 2026-10-27 00:43 · canonical https://reddit.com/r/LocalLLaMA/comments/1xg44nj/qwen3827b_jetson_thor_40_toks_at_fp16level