Jetson Thor vs RTX 5070 Ti: Unexpected LLM Inference Performance
retained text
Hi everyone, I have a question that might sound silly, but I haven't been able to find an answer. As far as I understand, memory bandwidth is the primary bottleneck for a model's inference performance. With my 5070 Ti (896 GB/s), if I run a Qwen3.8 27B Q4 model, I can roughly estimate the performance as 896 / 13.5 = \~66 tok/s, which is pretty much what I actually get (without MTP or DFlash). Recently, however, I had the chance to try a Jetson Thor (very similar to the DGX Spark). Based on its specs, I would expect 273 / 13.5 = \~20 tok/s, but I'm only getting around 12 tok/s. Even more surprisingly, with DFlash, which delivers exceptional performance gains on discrete GPUs, I'm barely getting a 2x speedup here, only slightly better than MTP. Does anyone know what might be causing this performance gap on this device? Thanks!
document fetched 2026-11-19 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1xzmfbu/jetson_thor_vs_rtx_5070_ti_unexpected_llm