How do you actually evaluate an agent harness (not the model) for autonomous app builds?
retained text
We’re a dev agency building internal harness around Claude Code. It takes a structured spec with numbered acceptance criteria and builds a small or medium web app up to a deployed preview. A human only steps in at a few checkpoints: spec, rules, final review. The question we keep arguing about internally: how do you objectively tell that harness version B is better than A, with the model held fixed? Our current plan: * **Frozen golden specs.** 5–10 of them, ideally replicas of past client projects. * **Holdout e2e/API tests.** Written by a human from the same spec and never visible to the agent. The main metric is “false-ready”: the agent’s own tests are green, but the holdout fails. * **Changeability.** After the build, a fixed baseline agent (plain Claude Code) applies 2–3 change requests, for example swapping the payment provider or adding a role. We measure regressions, diff size and tokens. Same executor each time, so differences reflect code quality. * **Blind pairwise review** of two runs against a rubric. * **At least 3 runs per config.** We report the median and pass\^k, not best-of. What I’d love to hear from people who’ve actually done this: 1. How do you build and maintain holdout sets without them leaking into the agent’s context? 2. Has anything measured code quality better than “how painful is the next change”? 3. Do you trust an LLM judge for pairwise review, and how did you calibrate it? 4. How many runs did you need before the variance settled, and how do you keep that affordable? 5. Which metrics turned out to be useless in practice? Please no AI-generated answers. I can ask Claude myself. I’m after what you’ve actually run in practice: what worked, what broke, and what you’d do differently. Vendor benchmarks don’t help much either. Happy to share our results once we have them.
document fetched 2026-11-18 00:22 · canonical https://reddit.com/r/ClaudeAI/comments/1xyhnzv/how_do_you_actually_evaluate_an_agent_harness_not