How is it that with the same model, the same prompt, temperature set to zero and a fixed seed, the answer is different because of Ollama's prompt cache? v2.0 - eddited
retained text
I'm testing LLMs to see how well they know Polish grammar. Why? Don't ask. Maybe I have too much free time, or maybe I'm some kind of freak who doesn't know what he's doing. ;) For this, I use my own tool, AI Bóbr, which is designed to test LLM performance in relation to the computer they're running on. And I noticed something strange. The model's first answer is sometimes different from the next four, even though the temperature is set to 0 and the seed is always 42. A little side note here, because one user, under this same post before I deleted it, pointed out that I'm doing Sisyphus' work by dealing with this subject. I think differently. And why did I delete the post? Believe it or not, I pressed the wrong button. So here I am posting it again. Anyway, back to the subject. Sitting there feeling defeated, just like now, because my written English is poor and algorithms on a forum about LLMs flag my posts as written by an LLM, I told myself: *"Man, stop wasting your life. Go make yourself some tea, turn off the computer. Live a little, go running, walk the dogs, play with your kids. In short, just leave this damn thing alone."* So that's what I decided to do... And then I sat down at the computer and ran another controlled experiment: * 3 models, * 10 model–task pairs, * 4 variants (streaming on/off × model kept loaded/unloaded), with 5 repetitions for each variant. **200 responses in total.** Observations: * In this sample, streaming had no effect on the answers. * Some answers changed depending on whether the prompt was reused from cache. * Bielik-Minitron 7B answered a question about the history of the Polish language correctly **5/5 times after unloading the model**, but got it wrong **5/5 times when the prompt was already cached**. What does this tell us? That with a fixed seed, if we don't control the state of the server, we can't accurately measure what we think we're measuring. What did I get out of this? I changed AI Bóbr in version 0.8.1. I added a controlled mode that unloads the model before each attempt, confirms that it has been unloaded, and records prompt-cache statistics. Now I'm preparing to run the benchmark of 11 models again under the same conditions. And maybe I'll take the dogs out for some fresh air. Tomorrow I definitely won't touch the computer, at least not until the evening. **And one more thing for everyone who reads my posts:** **1. Yes, I use AI to generate posts. I read them and make corrections if necessary.** **2. Yes, AI translates them into Polish and English.** **3. Yes, I'll keep doing it because it's convenient and fast.** **4. If someone doesn't like it — well, I'm not "Prince Polo" for eng. it will be "Mars bar", so not everyone has to like me.** **5. I'm here to share knowledge, not to show off the purity of my writing — even when it's generated by AI.**
document fetched 2026-11-07 00:40 · canonical https://reddit.com/r/LocalLLaMA/comments/1xpmwub/how_is_it_that_with_the_same_model_the_same