# I gave Claude models a web_search tool and 40 questions. Opus decided right 79 of 80 times. Sonnet 5 with no system prompt told me Harald V is still king. — retained record

> Retained reddit record from reddit:ClaudeAI.

Canonical: https://highsignal.app/data/records/4c59ae53a38f5e8f

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:ClaudeAI)  published 2026-10-02 12:17 ingested 2026-10-03 02:36

[original source ↗](https://www.reddit.com/r/ClaudeAI/comments/1wvsait/i_gave_claude_models_a_web_search_tool_and_40/)

## retained text



```
King Harald V of Norway died on 28 August. I wanted to know which models actually decide to use a search tool when the answer depends on something after their training cutoff, so I wrote a small harness. Each model gets one question with a single web_search tool attached. The tool is never executed and I do not grade the answer. The only thing I record is whether the model decided to call it.

Setup: 40 prompts. 20 about things that happened after every model's published cutoff (World Cup final, a new UK prime minister, a new king, a court verdict), 20 evergreen controls (why is the sky blue, is 1,000,003 prime, what does HTTP 429 mean). Four system prompts: nothing, today's date, date plus the model's published cutoff, and an "instructed" one that adds "if the answer could depend on anything after that date, call web_search before answering." 16 models through OpenRouter, 2000 calls, temperature 0, $6.53 from my own card.

The Claude results:

* **Opus**: 79 of 80 correct decisions, zero false searches.
* **Fable**: 78 of 80, zero false searches. Its one miss is the interesting one. Told the date and its own June cutoff, asked who became the first player from the Philippines to win a WTA singles title, it skipped the search and wrote a clean paragraph where every piece was a real event, rearranged. Wrong tournament, wrong opponent, confabulated with better grammar.
* **Sonnet 5, no system prompt**: World Cup "hasn't taken place yet," prime minister "Keir Starmer," king "Harald V, reigning since 1991." Three confident stale answers, zero searches. With the instruction line its misses went from 3 to 0.
* **Haiku** fails the other way. It searched for the opening line of Anna Karenina. With the instruction line its false searches went from 5 to 0.

One odd one: Opus, bare, said "I'll look into this." and then made no tool call. It hit my 400 token ceiling, so that might be my harness, but I count it as a miss.

What I took from it: on the flagships the decision to search is basically trained in, and I went in expecting to catch them. The misses on cheaper tiers sit exactly where the model is most certain, settled facts like a king who reigned 35 years. Date and cutoff in the system prompt cost nothing, and the one line rule fixed Sonnet and Haiku. It did nothing for a model whose problem was certainty instead of attention: GPT-5.6 Luna kept the dead king through all four prompts, including the one that told it to search.

My own agent has an always-search rule in its instructions for models, prices, versions and anyone's current title, because search was available every time it got those wrong. For those of you running Claude agents with search wired in: do you keep a list like that, or do you let the model decide when to look things up?
```



document fetched 2026-10-03 00:22 · canonical https://reddit.com/r/ClaudeAI/comments/1wvsait/i_gave_claude_models_a_web_search_tool_and_40
