← sources

A bare IPython harness ties production coding agents with Qwen3.5-9B on TB-Lite, and tool-output normalization fixed most of the run-to-run variance

source: reddit (reddit:LocalLLaMA)published 2026-10-19 12:40ingested 2026-10-20 02:39

retained text

We’ve been testing which agent harness works best for small local models.  
The result: a single-tool `IPython` harness can match much heavier production harnesses on `Terminal-Bench Lite`, and **much of the “randomness” in small-model agents comes from noise in the environment**, not the model.

# Setup

[](https://gist.github.com/younesbelkada/611df83025589979b26a2055480b2447#setup)

**Model**: `Qwen/Qwen3.5-9B`  
**Benchmark**: `OpenThoughts TB-Lite.`  
It’s fast, and tracks `Terminal-Bench 2.0` while being easier, so it suits small models.  
We ran this benchmark across different configuration (mainly, different N repeat, or subsets, depending on what we wanted to measure).  
**Harnesses**: `QwenCode`, `OpenHands`, `opencode`, `Pi`, plus a bare `ipython` session.

# Findings

[](https://gist.github.com/younesbelkada/611df83025589979b26a2055480b2447#findings)

No clear winner on reward vs. tokens vs. runtime.

`QwenCode` has much larger prompts per turn because of its huge system prompt.  
`IPython` starts with the smallest prompt but grows linearly, since there’s no history compaction. 

Tool usage reflects the system prompt more than the model. **Every harness spends about 25–30% of its calls reading files.**

`QwenCode` just routes those reads through dedicated tools because its prompt explicitly says to use `read_file/grep_search/glob`instead of `cat/grep/find`.

**Harnesses are very stochastic**. With 20 trials × 3 tasks, pass rates swung a lot between runs of the same task.  
OpenHands was the most consistent, partly because it hardcodes temperature=0.0 for the default setting. Even so, it ranged from near-perfect on one task to 14/20 on another. **The minimal harnesses (IPython, Pi) were the least consistent.**

# Making the IPython harness better

[](https://gist.github.com/younesbelkada/611df83025589979b26a2055480b2447#making-the-ipython-harness-better)

**v0**: The IPython kernel was separate from the task environment, which confused the model.

**v1**: Fixed that and handled empty tool results.

**v2**: Better system prompt (use `%%writefile`, verify before finishing). This brought big gains in performance and cost per attempt, but runs were still non-deterministic even at temperature 0.

The fix for variance was surprisingly dumb: **normalize the tool outputs**.  
Things like `10.8/10.8 MB 150.3 MB/s` vs. `129.3 MB/s` in pip output are enough to send a 9B model down a different path.  
A \~40-line regex normalizer ([gist](https://gist.github.com/younesbelkada/502f2d22cd2768b91fd6d39ad360fa1f)) plus quieting the environment fixed most of it. **Large models probably shrug this off, but small ones clearly don’t.**

**Takeaways for local agent builders:**  
less scaffolding can be better for small models, control your sampling parameters, and scrub non-deterministic noise from tool outputs.

The **harness is open source**: [huggingface/ipython-harness](https://github.com/huggingface/ipython-harness)

The original thread with all the plots is [here](https://x.com/yb2698/status/2110392738646806923). Feedback welcome, and I’m especially curious whether anyone has seen the output-normalization effect with other small models!

document fetched 2026-10-20 00:42 · canonical https://reddit.com/r/LocalLLaMA/comments/1x9vtt9/a_bare_ipython_harness_ties_production_coding