A bare IPython harness ties production coding agents with Qwen3.5-9B on TB-Lite, and tool-output normalization fixed most of the run-to-run variance
retained text
We’ve been testing which agent harness works best for small local models. The result: a single-tool `IPython` harness can match much heavier production harnesses on `Terminal-Bench Lite`, and **much of the “randomness” in small-model agents comes from noise in the environment**, not the model. # Setup [](https://gist.github.com/younesbelkada/611df83025589979b26a2055480b2447#setup) **Model**: `Qwen/Qwen3.5-9B` **Benchmark**: `OpenThoughts TB-Lite.` It’s fast, and tracks `Terminal-Bench 2.0` while being easier, so it suits small models. We ran this benchmark across different configuration (mainly, different N repeat, or subsets, depending on what we wanted to measure). **Harnesses**: `QwenCode`, `OpenHands`, `opencode`, `Pi`, plus a bare `ipython` session. # Findings [](https://gist.github.com/younesbelkada/611df83025589979b26a2055480b2447#findings) No clear winner on reward vs. tokens vs. runtime. `QwenCode` has much larger prompts per turn because of its huge system prompt. `IPython` starts with the smallest prompt but grows linearly, since there’s no history compaction. Tool usage reflects the system prompt more than the model. **Every harness spends about 25–30% of its calls reading files.** `QwenCode` just routes those reads through dedicated tools because its prompt explicitly says to use `read_file/grep_search/glob`instead of `cat/grep/find`. **Harnesses are very stochastic**. With 20 trials × 3 tasks, pass rates swung a lot between runs of the same task. OpenHands was the most consistent, partly because it hardcodes temperature=0.0 for the default setting. Even so, it ranged from near-perfect on one task to 14/20 on another. **The minimal harnesses (IPython, Pi) were the least consistent.** # Making the IPython harness better [](https://gist.github.com/younesbelkada/611df83025589979b26a2055480b2447#making-the-ipython-harness-better) **v0**: The IPython kernel was separate from the task environment, which confused the model. **v1**: Fixed that and handled empty tool results. **v2**: Better system prompt (use `%%writefile`, verify before finishing). This brought big gains in performance and cost per attempt, but runs were still non-deterministic even at temperature 0. The fix for variance was surprisingly dumb: **normalize the tool outputs**. Things like `10.8/10.8 MB 150.3 MB/s` vs. `129.3 MB/s` in pip output are enough to send a 9B model down a different path. A \~40-line regex normalizer ([gist](https://gist.github.com/younesbelkada/502f2d22cd2768b91fd6d39ad360fa1f)) plus quieting the environment fixed most of it. **Large models probably shrug this off, but small ones clearly don’t.** **Takeaways for local agent builders:** less scaffolding can be better for small models, control your sampling parameters, and scrub non-deterministic noise from tool outputs. The **harness is open source**: [huggingface/ipython-harness](https://github.com/huggingface/ipython-harness) The original thread with all the plots is [here](https://x.com/yb2698/status/2110392738646806923). Feedback welcome, and I’m especially curious whether anyone has seen the output-normalization effect with other small models!
document fetched 2026-10-20 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1x9vtt9/a_bare_ipython_harness_ties_production_coding