# How is it that with the same model, the same prompt, temperature set to zero and a fixed seed, the answer is different because of Ollama's prompt cache? v2.0 - eddited — retained record

> Retained reddit record from reddit:LocalLLaMA.

Canonical: https://highsignal.app/data/records/936f6ef942b76f5e

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:LocalLLaMA)  published 2026-11-06 20:13 ingested 2026-11-07 02:42

[original source ↗](https://www.reddit.com/r/LocalLLaMA/comments/1xpmwub/how_is_it_that_with_the_same_model_the_same/)

## retained text



```
I'm testing LLMs to see how well they know Polish grammar. Why? Don't ask. Maybe I have too much free time, or maybe I'm some kind of freak who doesn't know what he's doing. ;)

For this, I use my own tool, AI Bóbr, which is designed to test LLM performance in relation to the computer they're running on. And I noticed something strange.

The model's first answer is sometimes different from the next four, even though the temperature is set to 0 and the seed is always 42.

A little side note here, because one user, under this same post before I deleted it, pointed out that I'm doing Sisyphus' work by dealing with this subject. I think differently.

And why did I delete the post? Believe it or not, I pressed the wrong button. So here I am posting it again.

Anyway, back to the subject.

Sitting there feeling defeated, just like now, because my written English is poor and algorithms on a forum about LLMs flag my posts as written by an LLM, I told myself:

*"Man, stop wasting your life. Go make yourself some tea, turn off the computer. Live a little, go running, walk the dogs, play with your kids. In short, just leave this damn thing alone."*

So that's what I decided to do...

And then I sat down at the computer and ran another controlled experiment:

* 3 models,
* 10 model–task pairs,
* 4 variants (streaming on/off × model kept loaded/unloaded), with 5 repetitions for each variant.

**200 responses in total.**

Observations:

* In this sample, streaming had no effect on the answers.
* Some answers changed depending on whether the prompt was reused from cache.
* Bielik-Minitron 7B answered a question about the history of the Polish language correctly **5/5 times after unloading the model**, but got it wrong **5/5 times when the prompt was already cached**.

What does this tell us?

That with a fixed seed, if we don't control the state of the server, we can't accurately measure what we think we're measuring.

What did I get out of this?

I changed AI Bóbr in version 0.8.1.

I added a controlled mode that unloads the model before each attempt, confirms that it has been unloaded, and records prompt-cache statistics.

Now I'm preparing to run the benchmark of 11 models again under the same conditions. And maybe I'll take the dogs out for some fresh air. Tomorrow I definitely won't touch the computer, at least not until the evening.

**And one more thing for everyone who reads my posts:**

**1. Yes, I use AI to generate posts. I read them and make corrections if necessary.**

**2. Yes, AI translates them into Polish and English.**

**3. Yes, I'll keep doing it because it's convenient and fast.** 

**4. If someone doesn't like it — well, I'm not "Prince Polo" for eng. it will be "Mars bar", so not everyone has to like me.**

**5. I'm here to share knowledge, not to show off the purity of my writing — even when it's generated by AI.**
```



document fetched 2026-11-07 00:40 · canonical https://reddit.com/r/LocalLLaMA/comments/1xpmwub/how_is_it_that_with_the_same_model_the_same
