# I spent 9 months building local code search tool for Claude Code / 200 held-out SWE-rebench tasks: 5–11% cheaper, same no. of tasks solved. Open source (sweet-search) — retained record

> Retained reddit record from reddit:ClaudeAI.

Canonical: https://highsignal.app/data/records/e4b1b4a54f4bb41f

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:ClaudeAI)  published 2026-11-03 16:45 ingested 2026-11-04 02:37

[original source ↗](https://www.reddit.com/r/ClaudeAI/comments/1xmy1m0/i_spent_9_months_building_local_code_search_tool/)

## extract

Hi everybody, for the past 9 months I've been building sweet-search, a local code search tool for coding agents, so I wanted to try and share a mini-blog and my project.  
Instead of using native tools by reflex, the agent gets six fast search tools on a local index, plus a system prompt I hill-climbed for the lowest cost at unchanged answer quality.

The headline on 200 held-out real engineering tasks (SWE-rebench) across 5 harness + model combos vs native:  
\-> **5–33% cheaper** agentic runs (5-11% for Claude Code)  
\-> **same task resolution** (in total) (481 vs 481/1000)  
\->  **lower model time** for most setups -> faster agentic runs

This is the repo:  [https://github.com/mrsladoje/sweet-search](https://github.com/mrsladoje/sweet-search)

Setup is three easy commands:

    npm install -g sweet-search

    cd your-repo
    sweet-search init      # one-time: wires up Claude Code
                           # Codex: --codex · opencode: --opencode
                           # several: --claude --codex --opencode

    sweet-search index     # builds the index — GPU-accelerated where available

Everything is 100% local. Your code never leaves your machine.  
The embedding and late interaction models run 100% locally. The initial index is GPU-accelerated (Metal, CUDA, the Apple Neural Engine) or uses a tuned INT8 CPU path (optimized to have batches of maximum size that fit in L2 cache). After the first index, a low-footprint background daemon keeps it in sync with your working tree.

The agent calls the tools through its shell, so it doesn't bloat your context as MCP tool definitions (although there is an MCP server too, with `--mcp`, if you want one, however it is not benchmarked like the cli).  
`sweet-search uninstall` deletes everything.

I will TLDR you on my thought process, research and benchmarks.

*Where it started:*

I started with optimizing my own workflow for my job last year. I remember seeing [ruflo](https://github.com/ruvnet/ruflo) (formerly Claude Flow) and being very impressed with how far harnesses can be taken. I kept watching Claude Code and Codex grep, read a 2,000-line file, grep a synonym, read another file, spin up a subagent that does it all again, etc etc...  
Now, the harnesses and models have massively improved since (the introduction of caps for tool calls, reads target narrower spans now, ...) but I still think proper retrieval deserves a shot, despite the general consensus leaning towards agentic search (grep).

*Step 1: a better retriever (the "easy" part):*

I started where everyone starts. Hybrid search: BM25 keyword search, a code embedding model (*CodeRankEmbed*). The BM25 captured an F along the way (I use BM25F now). For fusion I use CCFusion with MMR, and obviously RRF fallback.  
Furthermore, I added a custom ANN kernel - a vector cascade (Hamming distance of direction bits → int8 rescore → float32 rescore) for cases with less than 150k vectors, and a really well designed HNSW graph for more vectors. This cascade beats top competitors like FAISS by Meta and USearch in accuracy and speed using their fastest setup (0.96-4.6x time needed to match 99% of our quality on M3 Max - more details in [README](https://github.com/mrsladoje/sweet-search#bench-engine-speed)). I will hopefully be publishing a paper on this topic in the coming days.  
What else makes my hybrid search tool (`ss-search`) great is its code graph based graph expansion. Adaptive 2-hop. Also IAR (Identifier Anchor Retrieval) and intent rerank.  
The most important detail is a ColBERT-style late-interaction rerank (using LateOn-Code) on my own Rust MaxSim kernel. This way we get cross-encoder accuracy at bi-encoder speed. I would also mention that my MaxSim kernel is pretty competitive (1.4-7.3x faster than NumPy, PyTorch and maxsim-cpu on M3 Max and x86 EPYC). This will also be a target of a publication in the coming period hopefully.  
Finally, it returns whole functions (chunks) and not just where the hit is, to avoid multiple turns for something that can be done in one.  
Chunking is done using cAST and tree-sitter (or regex fallback for some languages), and chunks are enriched with special context (this was ablated on, it differs per language).

To avoid fooling myself, I tuned only on a dev split of *GenCodeSearchNet* (stratified, seed 42) and looked at the other 2400 held-out queries in aggregate only:  **86.5 MRR@10**

On benchmarks I never tuned on, it's zero-shot SOTA on CoSQA (**65.4 MRR@10**; OpenAI text-embedding-3-large gets 55.4) and SOTA on *M2CRB* (**54.4 MRR@10**), the best published results I could find (rechecked October 2026).  
Every query ranks against the full corpus, not 99 sampled distractors.

*One more tool I built is my own grep variation -* `ss-grep`

On my benchmarks, it is 2.46× faster than ripgrep end-to-end (median 3.65 ms vs 7.51 ms) on 1,600 held-out queries.  
I could have made that number much more by cherry picking. But I wanna be honest, even at a price that it doesn't do my tool justice. The benchmark grep distribution is shaped like 19,762 real agent greps (by shaped I mean the flags used etc etc), over 13 repos, with the same hit counts. It's a sparse n-gram index (the idea comes from Cursor's fast regex search and GitHub's Blackbird), all in Rust inside the warm daemon.  
The speedup isn't that big because agents these days narrow their grep calls to subfolders when they used to grep across the whole repo before...  
That is why I like to say this all is a fight against windmills (Don Quixote reference). The big labs are spending billions of dollars RLing their agents to be efficient and I gotta try hard to keep up with them.  
Another perk of ss-grep is that it ranks hits differently. Agents often cut off the thing they need and would read in the next step with their head flag.  
My question was, could I optimize the ranking to rank the most likely hit higher near the top. So I set out to build just that and on my benchmarks, a variation of [Sainte-Laguë](https://en.wikipedia.org/wiki/Webster/Sainte-Lagu%C3%AB_method) and round-robin beat everything. The declaring line now shows up in 64% of calls, up from 34%.

*Things I learned the hard way in that phase:*

Upstream bugs can be silent. A padding mask in *candle* became *-Inf* in F16 on Metal, softmax turned every padded row into NaN, nothing crashed, and MRR fell to 25%. I rewrote the attention layer for both models (embedding and late interaction) on fused GPU kernels and fixed it there.  
I built a cross-encoder reranker too. It was 3× slower and did not beat MaxSim, so I don't ship it.

*Step 2: the agent didn't care (the actual hard part):*

Then the first real surprise: great retrieval scores did not make the agents cheaper. I gave Claude Code and Codex the tools, and they kept grepping. Six tools give the agent its power, but to no avail if it doesn't use them efficiently.

So I treated the agent's instructions as the thing to optimize. I evolved the tool prompt with GEPA: ran candidate prompts on Claude Code + Codex, kept the Pareto-best ones (cheapest at the best retrieval), let LLMs read the runs and write new candidates. Midway I realized that GEPA simply couldn't cut it (the models weren't so powerful back then), so I hand-tuned the prompt in collaboration with the models.  
Then I checked it on things the tuning never saw: 30 held-out questions (0.988), 8 OOD (unseen) languages (0.952), 2 unseen model families (MiMo 0.988, Qwen 0.980), and 6 rewordings of the prompt itself over 630 runs (same answers 0.95 on Claude Code, 0.93 on Codex).

After that came task completion. I had optimized the prompt for retrieval, but the agents kept stopping without finishing the tasks fully (it was following the prompt rules to stop too literally). That's when I knew the prompt needed further tuning, this time for task completion.  
I went through the failed traces of 400 dev tasks on Claude Code, Codex and opencode and changed my methodology. Instead of running GEPA I decided to "microsmoke" (so as to get the best results with the least money spent).  
What is microsmoking? I (or the agent) would write one new rule at a time, and kept a rule only if a micro-smoke on just a few target and control tasks said it helped. My experience is that current frontier models have a huge lack of real engineering judgement. They would require up to 20 tasks to make sure a rule helps, which is statistically better, but in reality 3 tasks with 2 control tasks should suffice - if you notice a mechanistic improvement, that is most likely enough. No need to go to lengths to be absolutely sure...

Lastly, each harness's own system prompt: some of its lines fight our rules (Claude Code's own prompt tells it to grep first). I removed only the lines that conflict (but carefully, sometimes I didn't if I realized that would break the cache), added a few focused ones (like packing independent calls into one turn), and microsmoked every edit. Lines that targeted a single measured waste survived. Generic wording never did.

*Step 3: benchmarking agents is hard*

During benchmarking, I ran into a few problems, one of which nearly made me publish garbage.

*Agent runs are noisy.*  
With the same task and same setup, the cost swings ±17–22% between identical runs, and I saw one task go 3/3 → 1/3 → 2/3 solved across identical reruns.  
To denoise and still make it budget and time-friendly the final task resolution benchmark has 200 tasks - enough to capture the run to run variance on the whole set. For the retrieval benchmark, I have a few reps, since it is cheaper and faster to run.

*My first task benchmark was contaminated. How did I fight back sneaky agents??*

When I audited the trajectories of the dev tasks, I found the agents getting to the answer through the back door: reading the gold patch from a cache file, pulling the real fix commit from public CDNs, running `git show` on the fix commit inside the test image, etc etc...  
On one held-out run (later demoted to dev), 13 of 16 native "wins" had help from the ground truth. I threw that run away.  
Now every rollout runs in its own mount/PID/network jail behind an allowlist proxy, so the agent can reach the model API and *nothing* else, and I count escape attempts instead of assuming they're zero.

*Prompt caching changes the answer.*  
Billed cost and cost without cache can point in opposite directions (see the Codex row below), so I report both.

***The benchmarks***  
(raw results, scripts and question sets are in the repo)

*Task completion:*

sweet-search vs the same model with only its built-in tools.  
200 held-out *SWE-rebench* tasks (200 repos, 16 languages, 125 multi-file), 1 rep per arm, graded by the official FAIL\_TO\_PASS + PASS\_TO\_PASS tests.  
Pinned versions: Claude Code 2.1.281, Codex CLI 0.159.2, opencode 1.18.4, sweet-search 2.9.0.

|Harness + model|Billed cost|Cost per solved task|Cost on tasks both arms solved|Solved (sweet / native)|
|:-|:-|:-|:-|:-|
|Claude Code + Sonnet 5.5 high|−10.8%|−15.6%|−16.3% (84 tasks)|92 / 87|
|Claude Code + Opus 5.5 medium|−5.4%|−7.9%|−5.7% (103, n.s.)|112 / 109|
|Claude Code + Opus 5.5 high|−7.4%|−2.3% (n.s.)|−4.0% (107, n.s.)|109 / 115|
|opencode + Sol 6.1 high|−18.6%|−14.7%|−11.4% (81)|84 / 88|
|Codex + Sol 6.1 high|−33.3%|−34.9%|−32.8% (77)|84 / 82|

(link in [README](https://github.com/mrsladoje/sweet-search#bench-task-completion))  

No solve difference is significant (*McNemar* exact test, p = 0.11 to 0.77), and pooled over all 5 setups it's 481 vs 481.  
The "both arms solved" column is there for the obvious objection: it isn't cheaper because it gives up early. On the 76 tasks that no rollout solved, the cost also fell in all 5 setups.  
(All runs used subscriptions, so the dollars are calculated list-price equivalents, not an invoice.)

*Code retrieval questions:*

200 held-out questions (103 easy + 97 hard) over 11 public repos in 11 languages that I never used in development.  
Multi-hop, call chains, completeness, decoys and negatives (the thing asked for does not exist). 3 reps per arm (6,000 rollouts, 0 failed), graded by the median of a 3-judge panel (DeepSeek, Gemini, MiniMax) against gold facts.  
Pre-registered, *Benjamini–Hochberg* corrected: 17 of 20 results survive.

|Harness + model|Accuracy|Billed cost|Cost without cache|Tool calls|
|:-|:-|:-|:-|:-|
|Claude Code + Sonnet 5.5 high|\+2.4% (0.899 vs 0.879)|−7.6%|−16.6%|−14.1%|
|Claude Code + Opus 5.5 medium|\+0.2% (n.s.)|−9.5%|−7.1%|−22.8%|
|Claude Code + Opus 5.5 high|−0.5% (n.s.)|−9.5%|−7.4%|−22.8%|
|opencode + Sol 6.1 high|−2.8% (0.869 vs 0.893)|−50.2%|−25.4%|−33.1%|
|Codex + Sol 6.1 high|−2.5% (0.858 vs 0.880)|−7.7%|\+22.9%|\+0.6% (n.s.)|

(link: [README](https://github.com/mrsladoje/sweet-search#bench-code-retrieval))  

Estimated model time per question (from tokens and turns): −8% to −14% on Claude Code, −34% on opencode, +8% on Codex.

*The exceptions, so you hear them from me first:*

*Sol 6.1 loses 2.5–2.8% accuracy on code questions.*  
I diagnosed it on dev questions only. Sol takes the rules too literally: the prompt says to stop once one file and symbol are confirmed, so Sol stops, then writes a short answer that leaves out facts it already found.  
In about 13 of the 14 worst questions, the gold fact was in a tool output but not in the answer. Sonnet and Opus read the same rules as guidance, not as something carved in stone - they see and feel the bigger picture.  
A Sol-specific fix did not hold up on fresh dev questions, so I didn't ship it.

*On Codex retrieval questions, the whole saving comes from the prompt cache*  
(+22.9% without it). sweet-search keeps the stock Codex prompt unchanged, so more of its input hits the cache.

*On opencode, the mechanistic number is −25.4%, not −50.2%.*  
sweet-search ships a cache-key plugin that stock opencode doesn't have, so cost without cache is the apples-to-apples comparison.

*Opus 5.5 high solved 6 fewer tasks*  
(109 vs 115, p = 0.11). Not significant, but it leans native. On medium reasoning, it solved 3 more... Sonnet solved 5 more... Maybe it is run-to-run variance.

*Where it helps most:*  
Go, TypeScript and Java tasks (−17 to −18%, mean of the 5 setups), feature requests (−18%, significant in all 5), and proving that something does not exist (−9% to −68% on retrieval questions, because two empty index searches settle it).  
Where it helps least: Python tasks (−9% mean), and PHP and Ruby questions sometimes cost more.

What the project lacks still:

\-> No native Windows. WSL2 works.  
\-> Deep AST parsing for 18 languages; 78 more get regex and fallback chunking.  
\-> One repo per index, so no org-wide multi-repo search.  
\-> On machines with 8 GB of RAM or less, it uses a smaller model that I have not benchmarked.

To summarize:

It's Apache-2.0, fully open source.

I would really appreciate feedback: repos where it does badly, harness requests, benchmarks you think I should run, or a hole in my methodology.  
All the raw results and scripts are in the repo, so please try to break the numbers.

P.S. If it saves your agent some tokens, a ⭐ helps other agents' humans find it lol. Because maybe grep isn't all you need… 🍬

## retained text (truncated)



```
Hi everybody, for the past 9 months I've been building sweet-search, a local code search tool for coding agents, so I wanted to try and share a mini-blog and my project.  
Instead of using native tools by reflex, the agent gets six fast search tools on a local index, plus a system prompt I hill-climbed for the lowest cost at unchanged answer quality.

The headline on 200 held-out real engineering tasks (SWE-rebench) across 5 harness + model combos vs native:  
\-> **5–33% cheaper** agentic runs (5-11% for Claude Code)  
\-> **same task resolution** (in total) (481 vs 481/1000)  
\->  **lower model time** for most setups -> faster agentic runs

  
This is the repo:  [https://github.com/mrsladoje/sweet-search](https://github.com/mrsladoje/sweet-search)

  
Setup is three easy commands:

    npm install -g sweet-search
    
    cd your-repo
    sweet-search init      # one-time: wires up Claude Code
                           # Codex: --codex · opencode: --opencode
                           # several: --claude --codex --opencode
    
    sweet-search index     # builds the index — GPU-accelerated where available

Everything is 100% local. Your code never leaves your machine.  
The embedding and late interaction models run 100% locally. The initial index is GPU-accelerated (Metal, CUDA, the Apple Neural Engine) or uses a tuned INT8 CPU path (optimized to have batches of maximum size that fit in L2 cache). After the first index, a low-footprint background daemon keeps it in sync with your working tree.

The agent calls the tools through its shell, so it doesn't bloat your context as MCP tool definitions (although there is an MCP server too, with `--mcp`, if you want one, however it is not benchmarked like the cli).  
`sweet-search uninstall` deletes everything.

  
I will TLDR you on my thought process, research and benchmarks.

  
*Where it started:*

I started with optimizing my own workflow for my job last year. I remember seeing [ruflo](https://github.com/ruvnet/ruflo) (formerly Claude Flow) and being very impressed with how far harnesses can be taken. I kept watching Claude Code and Codex grep, read a 2,000-line file, grep a synonym, read another file, spin up a subagent that does it all again, etc etc...  
Now, the harnesses and models have massively improved since (the introduction of caps for tool calls, reads target narrower spans now, ...) but I still think proper retrieval deserves a shot, despite the general consensus leaning towards agentic search (grep).

  
*Step 1: a better retriever (the "easy" part):*

I started where everyone starts. Hybrid search: BM25 keyword search, a code embedding model (*CodeRankEmbed*). The BM25 captured an F along the way (I use BM25F now). For fusion I use CCFusion with MMR, and obviously RRF fallback.  
Furthermore, I added a custom ANN kernel - a vector cascade (Hamming distance of direction bits → int8 rescore → float32 rescore) for cases with less than 150k vectors, and a really well designed HNSW graph for more vectors. This cascade beats top competitors like FAISS by Meta and USearch in accuracy and speed using their fastest setup (0.96-4.6x time needed to match 99% of our quality on M3 Max - more details in [README](https://github.com/mrsladoje/sweet-search#bench-engine-speed)). I will hopefully be publishing a paper on this topic in the coming days.  
What else makes my hybrid search tool (`ss-search`) great is its code graph based graph expansion. Adaptive 2-hop. Also IAR (Identifier Anchor Retrieval) and intent rerank.  
The most important detail is a ColBERT-style late-interaction rerank (using LateOn-Code) on my own Rust MaxSim kernel. This way we get cross-encoder accuracy at bi-encoder speed. I would also mention that my MaxSim kernel is pretty competitive (1.4-7.3x faster than NumPy, PyTorch and maxsim-cpu on M3 Max and x86 EPYC). This will also be a target of a publication in the coming period hopefully.  
Finally, it returns whole functions (chunks) and not just where the hit is, to avoid multiple turns for something that can be done in one.  
Chunking is done using cAST and tree-sitter (or regex fallback for some languages), and chunks are enriched with special context (this was ablated on, it differs per language).

To avoid fooling myself, I tuned only on a dev split of *GenCodeSearchNet* (stratified, seed 42) and looked at the other 2400 held-out queries in aggregate only:  **86.5 MRR@10**

On benchmarks I never tuned on, it's zero-shot SOTA on CoSQA (**65.4 MRR@10**; OpenAI text-embedding-3-large gets 55.4) and SOTA on *M2CRB* (**54.4 MRR@10**), the best published results I could find (rechecked October 2026).  
Every query ranks against the full corpus, not 99 sampled distractors.

*One more tool I built is my own grep variation -* `ss-grep`

On my benchmarks, it is 2.46× faster than ripgrep end-to-end (median 3.65 ms vs 7.51 ms) on 1,600 held-out queries.  
I could have made that number much more by cherry picking. But I wanna be honest, even at a price that it doesn't do my tool justice. The benchmark grep distribution is shaped like 19,762 real agent greps (by shaped I mean the flags used etc etc), over 13 repos, with the same hit counts. It's a sparse n-gram index (the idea comes from Cursor's fast regex search and GitHub's Blackbird), all in Rust inside the warm daemon.  
The speedup isn't that big because agents these days narrow their grep calls to subfolders when they used to grep across the whole repo before...  
That is why I like to say this all is a fight against windmills (Don Quixote reference). The big labs are spending billions of dollars RLing their agents to be efficient and I gotta try hard to keep up with them.  
Another perk of ss-grep is that it ranks hits differently. Agents often cut off the thing they need and would read in the next step with their head flag.  
My question was, could I optimize the ranking to rank the most likely hit higher near the top. So I set out to build just that and on my benchmarks, a variation of [Sainte-Laguë](https://en.wikipedia.org/wiki/Webster/Sainte-Lagu%C3%AB_method) and round-robin beat everything. The declaring line now shows up in 64% of calls, up from 34%.

*Things I learned the hard way in that phase:*

Upstream bugs can be silent. A padding mask in *candle* became *-Inf* in F16 on Metal, softmax turned every padded row into NaN, nothing crashed, and MRR fell to 25%. I rewrote the attention layer for both models (embedding and late interaction) on fused GPU kernels and fixed it there.  
I built a cross-encoder reranker too. It was 3× slower and did not beat MaxSim, so I don't ship it.

  
*Step 2: the agent didn't care (the actual hard part):*

Then the first real surprise: great retrieval scores did not make the agents cheaper. I gave Claude Code and Codex the tools, and they kept grepping. Six tools give the agent its power, but to no avail if it doesn't use them efficiently.

So I treated the agent's instructions as the thing to optimize. I evolved the tool prompt with GEPA: ran candidate prompts on Claude Code + Codex, kept the Pareto-best ones (cheapest at the best retrieval), let LLMs read the runs and write new candidates. Midway I realized that GEPA simply couldn't cut it (the models weren't so powerful back then), so I hand-tuned the prompt in collaboration with the models.  
Then I checked it on things the tuning never saw: 30 held-out questions (0.988), 8 OOD (unseen) languages (0.952), 2 unseen model families (MiMo 0.988, Qwen 0.980), and 6 rewordings of the prompt itself over 630 runs (same answers 0.95 on Claude Code, 0.93 on Codex).

After that came task completion. I had optimized the prompt for retrieval, but the agents kept stopping without finishing the tasks fully (it was following the prompt rules to stop too literally). That's when I knew the prompt needed further tuning, this time for task completion.  
I went through the failed traces of 400 dev tasks on Claude Code, Codex and opencode and changed my methodology. Instead of running GEPA I decided to "microsmoke" (so as to get the best results with the least money spent).  
What is microsmoking? I (or the agent) would write one new rule at a time, and kept a rule only if a micro-smoke on just a few target and control tasks said it helped. My experience is that current frontier models have a huge lack of real engineering judgement. They would require up to 20 tasks to make sure a rule helps, which is statistically better, but in reality 3 tasks with 2 control tasks should suffice - if you notice a mechanistic improvement, that is most likely enough. No need to go to lengths to be absolutely sure...

Lastly, each harness's own system prompt: some of its lines fight our rules (Claude Code's own prompt tells it to grep first). I removed only the lines that conflict (but carefully, sometimes I didn't if I realized that would break the cache), added a few focused ones (like packing independent calls into one turn), and microsmoked every edit. Lines that targeted a single measured waste survived. Generic wording never did.

  
*Step 3: benchmarking agents is hard*

During benchmarking, I ran into a few problems, one of which nearly made me publish garbage.

*Agent runs are noisy.*  
With the same task and same setup, the cost swings ±17–22% between identical runs, and I saw one task go 3/3 → 1/3 → 2/3 solved across identical reruns.  
To denoise and still make it budget and time-friendly the final task resolution benchmark has 200 tasks - enough to capture the run to run variance on the whole set. For the retrieval benchmark, I have a few reps, since it is cheaper and faster to run.

*My first task benchmark was contaminated. How did I fight back sneaky agents??*

When I audited the trajectories of the dev tasks, I found the agents getting to the answer through the back door: reading the gold patch from a cache file, pulling the real fix commit from public CDNs, running `git show` on the fix commit inside the test image, etc etc...  
On one held-out run (later demoted to dev), 13 of 16 native "wins" had help from the ground truth. I threw that run away.  
Now every rollout runs in its own mount/PID/network jail behind an allowlist proxy, so the agent can reach the model API and *nothing* else, and I count escape attempts instead of assuming they're zero.

*Prompt caching changes the answer.*  
Billed cost and cost without cache can point in opposite directions (see the Codex row below), so I report both.

  
***The benchmarks***  
(raw results, scripts and question sets are in the repo)

  
*Task completion:*

sweet-search vs the same model with only its built-in tools.  
200 held-out *SWE-rebench* tasks (200 repos, 16 languages, 125 multi-file), 1 rep per arm, graded by the official FAIL\_TO\_PASS + PASS\_TO\_PASS tests.  
Pinned versions: Claude Code 2.1.281, Codex CLI 0.159.2, opencode 1.18.4, sweet-search 2.9.0.

|Harness + model|Billed cost|Cost per solved task|Cost on tasks both arms solved|Solved (sweet / native)|
|:-|:-|:-|:-|:-|
|Claude Code + Sonnet 5.5 high|−10.8%|−15.6%|−16.3% (84 tasks)|92 / 87|
|Claude Code + Opus 5.5 medium|−5.4%|−7.9%|−5.7% (103, n.s.)|112 / 109|
|Claude Code + Opus 5.5 high|−7.4%|−2.3% (n.s.)|−4.0% (107, n.s.)|109 / 115|
|opencode + Sol 6.1 high|−18.6%|−14.7%|−11.4% (81)|84 / 88|
|Codex + Sol 6.1 high|−33.3%|−34.9%|−32.8% (77)|84 / 82|

(link in [README](https://github.com/mrsladoje/sweet-search#bench-task-completion))  
  
No solve difference is significant (*McNemar* exact test, p = 0.11 to 0.77), and pooled over all 5 setups it's 481 vs 481.  
The "both arms solved" column is there for the obvious objection: it isn't cheaper because it gives up early. On the 76 tasks that no rollout solved, the cost also fell in all 5 setups.  
(All runs used subscriptions, so the dollars are calculated list-price equivalents, not an invoice.)

  
*Code retrieval questions:*

200 held-out questions (103 easy + 97 hard) over 11 public repos in 11 languages that
```



document fetched 2026-11-04 00:21 · canonical https://reddit.com/r/ClaudeAI/comments/1xmy1m0/i_spent_9_months_building_local_code_search_tool
