← sources

We tested giving Claude Code a 450-line static prompt vs a self-learning harness (and measured a 38.4% drop in regression errors)

source: reddit (reddit:PromptEngineering)published 2026-09-23 09:33ingested 2026-09-24 02:40

retained text

Like a lot of people here, my team started out treating \`CLAUDE.md\` like a junk drawer. 

Every time Claude Code made an architectural mistake, failed to run our custom test runner, or imported a deprecated utility, we added another bullet point to root instructions. By month two, our instruction file was 450 lines long.

That was when the failure rate started climbing. The model would follow the first three rules, follow the last two rules, and completely ignore the 25 specific edge cases buried in the middle. Even worse, half the rules referenced file paths and database mocking conventions we had refactored three weeks earlier.

We ran a controlled benchmark across 40 multi-file refactoring tasks on our backend repository to compare two setups:

1. Setup A: The monolithic 450-line static \`CLAUDE.md\` file loaded on every session.

2. Setup B: A minimal 25-line root file paired with dynamic, on-demand skill distillation that auto-prunes stale rules.

Here is what we observed:

1. A 38.4% drop in regression errors: In Setup A, the model frequently hallucinated deprecated imports because old rules were competing with new codebase patterns. In Setup B, loading only task-relevant procedural instructions kept the working context clean.

2. Context window savings: Baseline input tokens per turn dropped from \~18,500 down to under 3,200. Because the agent was not paying full price to re-ingest 400 lines of static text on every command round-trip, average session token spend dropped by over 45%.

3. The silent killer is stale rules: The biggest drag on coding agent accuracy is not missing instructions; it is obsolete instructions. When a project updates a library API, an unpruned rule file actively trains the model to write code that fails CI.

To make Setup B reproducible, we open-sourced our internal tool as autoharness:

https://github.com/tigerless-labs/autoharness

How it works under the hood:

\- Session distillation: When you complete a complex debugging or setup task in the terminal, it extracts the verified command sequence and packages it into a modular \`SKILL.md\` file.

\- On-demand loading: Skills only enter the prompt context when that specific task is invoked.

\- Automatic pruning: If a skill has not been triggered across recent sessions or references files that no longer exist, autoharness prunes it so your harness stays lean.

\- Zero daemons: Pure Python standard library with no background services, no local vector databases, and no telemetry.

If you are using Claude Code on larger codebases, what does your rule maintenance loop look like? How many lines is your current \`CLAUDE.md\` before model recall starts degrading?

document fetched 2026-09-24 00:42 · canonical https://reddit.com/r/PromptEngineering/comments/1wo18s1/we_tested_giving_claude_code_a_450line_static