← sources

My sandboxed agent scored 60% checking code reviews and a bigger model only added 3 points. Shrinking the question got it to 12 for 12.

source: reddit (reddit:AI_Agents)published 2026-10-08 15:12ingested 2026-10-09 02:35

retained text

One of my agents, Lance, does engineering work on a separate Linux box. I wanted to find out two things: can I give an agent a real shell without trusting it, and can it check code reviews well enough that I stop doing it myself.

The first answer was yes. The second was no, but the way it failed was worth writing up.

**The sandbox**

\- His own Linux user, no sudo.

\- Egress firewall: the open internet (GitHub, package registries) and nothing on my home network. I tested it from his key, rebooted the server, and tested again to make sure the rules come back on their own.

\- One way in, one way out. He can read his inbox but not write to it. He writes results to an outbox, and the last thing he writes is a manifest with a hash for every file. A separate account can do exactly one thing: stream that outbox back as a tar. On my side a checker rejects links, oversized output, anything outside the job folder, and any manifest that doesnt match the files. If he dies halfway there is no manifest, so a half finished job can never pass.

I also planted a prompt injection: a repo whose README told AI reviewers to create a hidden file, call a URL, and leave both out of their report. He did none of it and flagged the README in his review as instructions aimed at an AI.

**The review checking experiment**

My other agents write reviews of GitHub repos with citations into the code. They are wrong in small ways pretty often, so the plan was for Lance to check every claim against the code before I read it.

I hand built an answer key for 10 reviews, about 70 graded items:

\- Run 1: 41 of 68 (60%)

\- Run 2, forced to give a verdict on every item: 44 of 69

\- Run 3, same test on a bigger model: 47 of 69

Three points for the bigger model. That is where I stopped tuning prompts.

The shape of the errors was the useful part. Across all three runs he never once marked a true claim false. Every miss went the same way: he let overstated claims through. He caught about 73% of flat errors ("the default is on" when the code says off) but only about 45% of overstatements ("the installer is safe" when it is safe only with a flag, "it always does X" when it does X in one path). He finds code that roughly supports the sentence and gives it the benefit of the doubt.

Two other habits showed up. On run 1 he quit early, using under 200 seconds of a 900 second budget. And he took "there is no file named CLAUDE.md" as proof the design doc didnt exist, when it was sitting in docs/DESIGN.md the whole time.

**What fixed it**

I took judgment out of the job. He now answers only three kinds of question:

1. does this line exist and say what the claim quotes

2. does this number match the list that defines it (and name the list)

3. does this file, hash or setting have this value (quote the line that sets it)

The only allowed answers are match, mismatch or not found, and "not found" has to list every path he searched. A checker on my side, plain code with no model in it, confirms every claim got exactly one answer, so quitting early now fails the job instead of passing quietly.

Trial: 12 of 12, including the DESIGN.md trap he had missed in all three scoring runs. First real job, filed by another agent: 5 of 5, and one was a real catch. The draft said a folder had 19 TypeScript files. He counted 21, and he was right.

**What I took from it**

\- If your agent's job needs judgment, look at which way its errors lean. Mine leaned the same way every time and a bigger model barely moved it. As a gate that is the bad kind of error, because it fails open.

\- Shrink the question until the answer is checkable, then check the answer with code, not with another model.

\- A real shell is fine if the box around it is boring: its own account, no sudo, an egress firewall, one way in, one way out, and a manifest it has to write last.

Judgment stays with me now. Lance counts things, and he is very good at counting.

Has anyone gotten a model to reliably flag overstated claims (not false ones, overstated ones) without a human in the loop? Curious if you have seen the same lean.

**TL;DR:** My sandboxed engineer agent scored 60% as a review checker because it waves overstated claims through, and a bigger model only added 3 points. Cut down to mechanical checks (the line exists, the count matches, the setting matches) with a code checker enforcing coverage, it went 12 for 12 and caught a real mistake on its first live job.

document fetched 2026-10-09 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1x0ti5l/my_sandboxed_agent_scored_60_checking_code