← sources

What actually took the time in an AI agent setup for a legacy ERP/WMS: knowledge, checks and approvals

source: reddit (reddit:AI_Agents)published 2026-10-27 08:05ingested 2026-10-28 02:35

retained text

Most coding-agent demos are an issue in, a PR out. That works because GitHub supplies the rest: the issue describes the work, tests check it, a reviewer approves it, and merging deploys it.

I’ve spent the past few months building agents to support a customised IBM Sterling WMS (a warehouse product), and none of those assumptions held:

* **Behaviour isn’t only in the code.** It’s split between code, configuration stored in each installation’s database, and a vendor product we can’t change.
* **There’s no test suite.** The toolchain is Java 6, and some configuration only exists on an installed system.
* **A person deploys,** in an agreed change window.

So the work doesn’t disappear; it moves. Here’s what I learned, without client code or tickets:

**1. Agents follow what’s enforced, not what’s asked.** The prompt told the agent many times to read our conventions before writing anything. It didn’t. In one run, the same agent called a tool the system required 4 times and a tool the prompt asked for 0 times. So we declared tool order in config (`requires:`): calling a tool too early is refused with a message saying what to call first.

**2. But that doesn’t work on a full coding agent.** Claude Code has its own file tools, so it never touched ours, and a rule on a tool it never calls just gives a false sense of safety. On harness agents we only put rules on tools with no built-in alternative (data model, docs, live config). For everything else, we check what the agent *returns*, not how it got there.

**3. Ask for evidence in the output.** For every file it writes, the coding agent must name the existing file it modelled it on, and a script flags any file without one. This worked where restricting its tools didn’t.

**4. Never accept “done” as proof.** One run lost its tools halfway through and still reported success. Designs now go through three checks in order:

* a JSON schema check;
* scripted fact checks (does every service and file the design names actually exist in the installation’s config?);
* an LLM reviewer, which only runs if the scripted checks pass.

A person approves last, and can’t be the author. Tests are written from the approved spec, not from the code, and JUnit decides pass or fail. The test-run step has no agent at all.

**5. Pick the executor per step, and measure.** A cheap open model (Kimi K2.5) handled triage’s short form fine. On the design step (22 fields plus tool calls) it produced a valid design in only 2 of 7 runs, so design moved to Claude Code. Most of the savings didn’t come from model choice, though:

* scripts wherever there’s a definite answer;
* skipping steps that have nothing to do;
* cheap checks before expensive ones;
* no re-approval when the output hash hasn’t changed.

**6. Knowledge only counts if something delivers it.** An audit found six places where knowledge was set up but nothing ever loaded it. One design run rebuilt an approach we had already recorded as wrong ($3.63) while the right fact sat unused in storage. With the fact actually delivered, the next run used it and cost $1.75.

**7. Don’t let agents write their own memory.** It looks great in a demo and slowly goes wrong: guesses become facts, and agents keep confirming their own earlier guesses. Only people write facts, at the points where something stops being a guess (an accepted diagnosis, a closed ticket).

**8. Move human decisions earlier.** People approve the triage and the spec, not just the final diff. Fixing a misunderstanding in the spec means rewriting the spec; fixing it after the code is written means another change, another review and another integration test cycle. Each kind of reviewer objection (“that’s not how we build it here”, “that file doesn’t exist”) became a convention card or a scripted check, so approvals got faster without going away.

**The open problem is people.** As volume grows, approvals become the bottleneck before the agents do. Next we’re trying shadow mode: people still decide, while we record how often they agree with the automated checks.

Full write-up, with the architecture, the convention-card list and how rework is handled, link in the comments.

Disclosure: the agents run on SwarmKit, an open-source runtime I maintain. The lessons above don’t depend on it.

Curious whether others doing agent work on legacy enterprise systems have hit the same thing: has the review or approval step become your bottleneck too?

document fetched 2026-10-28 00:20 · canonical https://reddit.com/r/AI_Agents/comments/1xgmles/what_actually_took_the_time_in_an_ai_agent_setup