# Model swap came back 59% cheaper and 75% faster. I still wouldn’t ship it. — retained record

> Retained reddit record from reddit:AI_Agents.

Canonical: https://highsignal.app/data/records/ea61e82a0bc95cc2

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:AI_Agents)  published 2026-11-02 22:26 ingested 2026-11-03 02:36

[original source ↗](https://www.reddit.com/r/AI_Agents/comments/1xmbkzc/model_swap_came_back_59_cheaper_and_75_faster_i/)

## retained text



```
Disclosure: I’m the founder of EvalShift, and this came from one of its reports.

Cost and latency can’t tell you a model swap is safe. I ran a migration where both improved a lot, with zero failed calls, and the agent still changed enough that I’d block it.

I replayed 16 real examples against the old and new model. Cost went from $0.1799 to $0.0741. Average latency went from 3.1 s to 773 ms. No failed or truncated calls.

Then I looked at what the agent actually did:

* Someone asked for an opinion on working late. The old model answered directly. The new one called search\_web and returned no text.
* An action was reported as completed, but the tool was never called.
* A request got routed to a different tool.
* A tool call appeared where the old model had just answered.

Tool selection diverged 25% of the time against my 10% limit. Equivalence was 61.4% against a 75% floor. The funny part is the overall regression rate was 13.6% and passed its budget, so one blended score could have let this through.

The fake completion is the one I’d worry about most. It reads like a success, and if you only diff text, you won’t see anything wrong.

So now I compare traces: which tools fired, with what args, in what order, on the same inputs for both models. I also set the limits before looking at the results, because a 59% cost cut makes it very easy to talk yourself into accepting a regression.

16 examples is a small suite. It was enough to catch this, but I’d want more before trusting a pass.
```



document fetched 2026-11-03 00:18 · canonical https://reddit.com/r/AI_Agents/comments/1xmbkzc/model_swap_came_back_59_cheaper_and_75_faster_i
