# How are you benchmarking coding agents without accidentally benchmarking the harness? — retained record

> Retained reddit record from reddit:ClaudeCode.

Canonical: https://highsignal.app/data/records/7cceaf371af42450

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:ClaudeCode)  published 2026-10-07 17:17 ingested 2026-10-08 02:41

[original source ↗](https://www.reddit.com/r/ClaudeCode/comments/1x02e3s/how_are_you_benchmarking_coding_agents_without/)

## retained text



```
I’ve been trying to compare a couple model/provider combinations for coding work and I’m realizing how hard it is to keep the comparison clean. If I give two setups the same task, one agent might make 12 model calls while another makes 25. One reads half the repo, another grabs three files. One runs tests repeatedly and another waits until the end.

So even if provider A serves the model faster, provider B can still finish first because the agent took a completely different path. How are you guys doing meaningful performance comparisons here? Do you lock down the harness and only change the API provider, or do you just care about total wall-clock time in the end?


```



document fetched 2026-10-08 00:41 · canonical https://reddit.com/r/ClaudeCode/comments/1x02e3s/how_are_you_benchmarking_coding_agents_without
