Why Temperature 0 Isn't Deterministic in LLM Inference

retained text

In today’s newsletter:
- A voice agent that gets the details right.
- Why temperature 0 isn’t deterministic in LLM inference.
A voice agent that gets the details right
Speech accuracy matters on both sides of an LLM. Speech-to-text must preserve what the caller says, while text-to-speech must preserve the response the LLM generates.
That becomes especially important with medication names, dosages, appointment times, and alphanumeric reference codes.
Deepgram's Flux TTS handles the outbound speech layer. It recorded a 3.4% word error rate in an evaluation containing drug names, tracking numbers, technical strings, and alphanumerics.
We tested it through a pet-hospital conversation about a dog’s prescription refill. The agent retained the medication, dosage, prescription number, and a collection time that changed during the call.
The details remained correct, the voice sounded natural, and the interaction felt like a call I would be comfortable having with an actual pet hospital.
Thanks to Deepgram for partnering today.
Why temperature 0 isn’t deterministic in LLM inference
At temperature 0, the model does not sample. At every step, it picks the token with the highest logit, so the same prompt should give the same output every time.
In practice, it doesn’t.
Thinking Machines sent “Tell me about Richard Feynman” to Qwen3-235B 1,000 times at temperature 0 and got 80 different completions.
Picking the highest logit is perfectly deterministic. The variation comes from computing the logits themselves, which happens from scratch on every request through billions of additions.
If those numbers come out slightly different, argmax can pick a different token.
Let’s understand why the logits change.
1) The order of addition changes the result
GPUs store numbers with limited precision. fp16 keeps only 11 significant bits (10 stored mantissa bits plus 1 implicit bit).
Between 1024 and 2048, that leaves no room for anything after the decimal point. Representable values are spaced exactly 1 apart, so 1024.4 gets rounded back to 1024.
Now consider 1024 + 0.4 + 0.4 in two orders. The diagram below shows both.
Left to right, 1024 + 0.4 rounds back to 1024, and adding 0.4 again rounds back to 1024 once more. The answer is 1024.
If the small numbers go first, 0.4 + 0.4 gives 0.8, and 1024 + 0.8 rounds up to 1025. The answer is 1025.
You can verify this yourself:
import numpy as np
a, b = np.float16(1024), np.float16(0.4)
print((a + b) + b)   # 1024.0
print(a + (b + b))   # 1025.0
In other words, floating-point addition is not associative. An LLM performs billions of such additions in every forward pass, so if their order changes, the logits change slightly too.
2) What changes the order?
The popular answer is that parallel GPU threads finish in an unpredictable order.
That isn’t the main reason. The kernels in a typical forward pass compute each sum in a fixed order, and the same matrix multiplication on the same inputs gives bit-for-bit identical results every time you run it.
So the forward pass itself is run-to-run deterministic. Something else must be changing between requests.
3) The real reason is the other requests on the server
An inference server groups requests from many users into a batch and runs them together.
The kernels decide how to split their work based on the batch size. With a large batch, every GPU core is busy, so each core computes its sum in one go.
With a small batch, many cores would sit idle. So the kernel cuts each long sum into chunks, spreads them across cores, and combines the partial results at the end. This shows up in matmuls (split-K), in RMSNorm, and in attention during decoding (split-KV, as in FlashDecoding).
A different split means a different order of addition. The batch size, in turn, depends on how many other people are using the server at that moment.
Since you have no control over that, the output looks random from your side. Thinking Machines call the missing property batch invariance: a request’s numbers should not depend on what else is in the batch.
4) How a tiny difference changes the entire answer
A difference in the fourth decimal place sounds harmless, and most of the time it is.
Consider two runs of the same prompt. The numbers below are illustrative, but the mechanism is real.
→ In a batch of 8, the logits come out as “Queens” = 12.3417 and “New” = 12.3415, so argmax picks “Queens”.
→ In a batch of 32, the rounding works out slightly differently, giving “Queens” = 12.3414 and “New” = 12.3416, so argmax picks “New”.
The output only flips at such near ties. Most of the time, the top token wins by a comfortable margin and rounding noise can’t touch it.
The problem is what happens after a flip. Generation is autoregressive, so every token after that point is computed from a different prefix.
That’s exactly what Thinking Machines saw. In the Feynman test, all 1,000 runs matched for the first 102 tokens. At token 103, 992 runs wrote “Queens, New York” while 8 wrote “New York City”. From there, the completions diverged into 80 unique outputs.
5) The fix and its cost
The fix follows directly from the cause. Every kernel should split its work the same way regardless of batch size, even if that leaves some cores idle on small batches.
For attention, Thinking Machines fix the split size rather than the number of splits, so a given token always lands in the same chunk. They also update the KV cache before the attention kernel runs, so cached and freshly computed tokens are reduced in the same order.
With these batch-invariant kernels, all 1,000 Feynman runs were identical.
The cost is speed. On Qwen3-8B with 1,000 requests, default vLLM took 26 seconds. The first deterministic version took 55 seconds, and an improved attention kernel brought it down to 42 seconds.
Both major open-source engines now ship this:
- vLLM: the VLLM_BATCH_INVARIANT=1 environment variable (beta).
- SGLang: the --enable-deterministic-inference flag, with a reported average slowdown of 34.35%.
If you just chat with a model, none of this matters much. It matters a lot for evals that need to reproduce, for replaying a bad production output while debugging, and for RL, where the sampler and the trainer must compute the same probabilities for training to stay truly on-policy.
So temperature 0 removes the sampling randomness, but it doesn’t remove the batch.
The visual below depicts the entire process.
If you’re shipping LLMs to production, issues like this one rarely show up in a demo. They show up later as evals that won’t reproduce, latency that spikes under load, or a serving bill that grows faster than traffic.
That’s the gap the LLMOps course fills. It walks you through the full lifecycle, from how LLMs work under the hood to prompting, context, evaluation, fine-tuning, inference optimization, and serving, so you can reason about these problems before they hit your users.
If you want to learn the full LLMOps lifecycle, we have covered every bit of it in the LLMOps course, starting from fundamentals to production:
- Read Part 1 on fundamentals of LLMOps here →
- Read Part 2 on understanding the core building blocks of LLMs →
- Read Part 3 on the key components of LLMs, focusing on the attention mechanism, architectures like transformers and mixture-of-experts, and the fundamentals of pretraining and fine-tuning →
- Read Part 4 on decoding strategies, generation parameters, best practices, and the broader lifecycle of LLM-based applications →
- Read Part 5 on context + prompt engineering from a system perspective, in-context learning, types of prompts, and different prompting techniques →
- Read Part 6 on prompt versioning, defensive prompting, and techniques like verbalized sampling, role prompting, and more →
- Read Part 7 on context engineering, covering context types, context construction principles, and retrieval-centric techniques for building high-signal inputs →
- Read Part 8 on memory, dynamic, and temporal context in LLM systems, covering short and long-term memory, dynamic context injection, and common failure modes in agentic applications →
- Read Part 9 on evaluation methods and approaches for LLM-based applications, primarily focusing on building a strong understanding of the fundamental concepts →
- Read Part 10 on evaluation benchmarks in LLM applications, with task-specific methodologies, and the core tooling for evaluation of LLM apps →
- Read Part 11 on evaluation of multi-turn systems, tool use evaluations, tracing, and red teaming →
- Read Part 12 on LLM fine-tuning, parameter-efficient methods like LoRA and QLoRA, and alignment techniques such as RLHF, DPO, and GRPO →
- Read Part 13 on LLM inference optimization, KV caching, PagedAttention, FlashAttention, speculative decoding, and model parallelism →
- Read Part 14 on the fundamentals of LLM serving, including API-based access, inference with vLLM, and practical decisions →
👉 Over to you: Have you ever had an eval score move between two identical runs?
Good day!

document fetched 2026-10-27 02:58