I tested 4 Claude Code token savers on real tasks with hidden tests. Only one lowered the bill, and only on big jobs
retained text
I wanted to know if token-saving plugins actually lower the bill, so I ran them against plain Claude Code on two real tasks and graded everything with hidden tests the agent never saw. **Setup** - Claude Code 2.1.289, Sonnet 5.5, headless (`claude -p`), same flags every run - Fresh Docker container per run, nothing carried over - Task 1: a hard bug in humanize (precisedelta), 66 hidden tests - Task 2: an 8-file feature in WeasyPrint (box-shadow), 4,292 hidden tests - 3 runs per tool per task, 6 for the bug baseline and caveman - Cost repriced from the token counts at Sonnet 5.5 API list prices **Median cost vs plain Claude Code** | tool | small bug | 8-file feature | |---|---|---| | ponytail v5.3.2 | +20% | -34% | | caveman v4.0.1 | -3% | -3% | | superpowers v7.0.0 | +32% | +2% | | answer-me-with-html v0.5.5 | never activated (0/6) | never activated | **Output tokens (what the agent writes)** - ponytail: -38% on the feature - caveman: -21% on the bug - superpowers: +20% on the bug **Hidden test scores** stayed basically the same for all of them. Superpowers passed 1 more test on the bug (63/66 vs 62/66). **What I took from it** Caveman cut output by 21% and the bill barely moved. Output is only part of the cost. Most of it is the agent re-reading context every turn (cache reads and writes), and none of these tools touch that much. Ponytail was the only one that cut reads too, and only on the long task, which is where it saved 34%. On a quick bug it added overhead and cost more. **Caveats** - Small sample, Sonnet only. Caveman's -3% is within noise. - Caveman's hook stays off in headless mode unless you set `CAVEMAN_DEFAULT_MODE=caveman`, so I set it. - answer-me-with-html showed up in the skill list but never got invoked in 6 runs. It might do better if you call it explicitly. - Costs are API-rate estimates from token counts, not a real invoice. Total spend for 36 runs: about $10.58. Repos: - ponytail: github.com/DietrichGebert/ponytail - caveman: github.com/JuliusBrussee/caveman - superpowers: github.com/obra/superpowers - answer-me-with-html: github.com/QingYunA/answer-me-with-html Happy to share the per-run numbers if anyone wants them. Running a second round now on a 3D build task to see if it holds on a different kind of job.
document fetched 2026-10-24 10:41 · canonical https://reddit.com/r/ClaudeCode/comments/1xdqtfa/i_tested_4_claude_code_token_savers_on_real_tasks