I ran the same browser job 216 times with Claude Code and Codex. A script beat every agent run, and the best browser tool depended on the agent
retained text
I wanted to figure out what the best browsing tool for agents is, so I had Claude Code (Opus 5.5) and Codex (GPT 6 Astra) do the same job in a business web app. They had to pick the right department, open a report in a new tab and check it, read its 500 rows and save them, close the tab, and go back to the start page. I tested agent-browser, Playwright and Chrome DevTools, each over its CLI with its skill, its CLI without the skill, and MCP. Each setup got 12 runs, split across macOS, WSL and Windows, always in a fresh session. All 216 passed. Since every run passed, I'd choose by token use and speed, and the answer depended on the agent. These are medians per run: | Agent | Tool | Connection | Input tokens | Cost | Time | |---|---|---|---|---|---| | Claude Code | agent-browser | CLI + skill | 652K | $0.48 | 104s | | Claude Code | agent-browser | CLI, no skill | 304K | $0.28 | 96s | | Claude Code | agent-browser | MCP | 1.59M | $1.02 | 106s | | Claude Code | Playwright | CLI + skill | 332K | $0.32 | 90s | | Claude Code | Playwright | CLI, no skill | 353K | $0.33 | 99s | | Claude Code | Playwright | MCP | 513K | $0.41 | 98s | | Claude Code | Chrome DevTools | CLI + skill | 341K | $0.32 | 97s | | Claude Code | Chrome DevTools | CLI, no skill | 348K | $0.32 | 98s | | Claude Code | Chrome DevTools | MCP | 517K | $0.40 | 72s | | Codex | agent-browser | CLI + skill | 426K | $0.91 | 91s | | Codex | agent-browser | CLI, no skill | 289K | $0.68 | 78s | | Codex | agent-browser | MCP | 414K | $0.89 | 72s | | Codex | Playwright | CLI + skill | 318K | $0.80 | 102s | | Codex | Playwright | CLI, no skill | 243K | $0.66 | 91s | | Codex | Playwright | MCP | 261K | $0.62 | 88s | | Codex | Chrome DevTools | CLI + skill | 197K | $0.53 | 75s | | Codex | Chrome DevTools | CLI, no skill | 233K | $0.58 | 86s | | Codex | Chrome DevTools | MCP | 238K | $0.60 | 81s | With Codex, Chrome DevTools' CLI with its skill did well on both. With Claude Code there was a trade-off. Each model ran in only one harness, so I can't separate the model's effect from the harness's. The gap between agents and a script was far bigger. Claude Code wrote a Python script over CDP from the notes of an earlier session. It took a median 1.5s over 258 runs, and even its slowest run (10.3s) was faster than the fastest agent run (59s). Those times leave out writing it. When I broke the script on purpose, the session running it finished the job itself and proposed fixes, which went into the script. So for a job you repeat, I'd let the agent work it out once, then run the script and bring the agent back when it breaks. This is one job in one web app at fixed versions, and another job could order the tools differently. The app's name and the report's contents are private, so you can't rerun it exactly. Costs are what Claude Code reports at API prices and what Codex's tokens come to at list prices, which differ from what a subscription charges. A row can mix values from different runs because each measure's median is taken on its own. The write-up with every run is in the comments. If you run agents on repeated jobs, do you keep the agent in the loop or hand the work to a script?
document fetched 2026-11-03 00:18 · canonical https://reddit.com/r/AI_Agents/comments/1xm0ac4/i_ran_the_same_browser_job_216_times_with_claude