PSA if you build agents on a local model: my harness was silently breaking its own KV cache (43% of a run)
retained text
​ \*My setup: Ollama · Swift 1.5 (Qwen3.8 27B) IQ3\_S + MTP · 128K context · RTX 3090 · Windows Two of us are building a local agent with Claude Code. It drives a real Chrome, reads pages, and writes the results into Excel. It works. It was also slow, and for a few weeks we kept finding reasons. **Suspect 1:** things that change every run sat at the top of the prompt. So the first call of every run re-read the whole thing. We moved them down. Fixed, but runs were still slow. **Suspect 2:** switching thinking on and off mid-run. Out of 955 logged calls, the 20 where thinking flipped waited a median 47.3s for the first token. The 852 where it didn't: 1.9s. Real, but rare. **Suspect 3**: a logging feature we'd left on. Turned it off, restarted, one prefill went from 32s to 3.7s. One run, so we never knew if it was the feature or the restart. **Suspect 4**: narrowing the tool list mid-run. The tool list sits before everything else, so changing it re-reads it all. One call took 93s. Fixed. After that we settled on "it's a 27B on a consumer card, it's just slow." We even wrote that in the README. Then a commenter on my last post suggested splitting model time from tool-wait time. So I did, on three runs from that night: \- Tool time (mostly opening pages): 7 to 10% of each run, about 2s per page. \- Calls where the history was unchanged: prefill 1 to 2s (11 to 14s when a big page got appended). \- Calls where something earlier in the history had changed: 25 to 40s each. In a Reddit research run, 15 of 38 calls were the second kind. 552s of a 1,274s run, 43%. Nothing earlier in the history should change. Pages that were already read are just text in the log. So what was changing it? Us. To stay under the context budget, the harness shrinks old tool results. But it re-picked which ones to shrink on every call, so the cut line slid forward by one message each time, and the cache broke from that point on. Every time. The fix: shrink once and remember it. Send the same shrunk version on every later call. Only shrink again when the history actually overflows, and then in one big batch. The history becomes append-only. Replaying the same logs offline, the rewrites went from 15 to 5 and the re-read text dropped 61%. Then the real test: the same two tasks, 3 runs each before and after, same PC. Prefill per run went from 327s to 196s, and a whole run from 721s to 541s (−25%), with the same plan completion and the same number of rows filled. Only 3 runs per side, so read it as a direction, not a benchmark. If you run an agent on a local model and it feels slow, log the prefill time of every call and check whether the start of your prompt is identical to the last call. Ours wasn't, and it was our own code. Next thing I'm chasing: step count. Reading 11 GitHub repos took 56 steps, 21 of them just opening pages, and every step pays for prefill, thinking and writing. Sites with an API (GitHub, Hugging Face) don't need a browser at all, and most pages only have two or three fields we actually use. I'll post the numbers when I have them. Has anyone else found their harness quietly breaking the cache? Where was it in yours?
document fetched 2026-10-25 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xe90df/psa_if_you_build_agents_on_a_local_model_my