← sources

How do you test that an agent updates the right record when names collide?

source: reddit (reddit:AI_Agents)published 2026-10-12 01:25ingested 2026-10-13 02:36

retained text

Here's a small test case I'd want in an agent evaluation. All names and records are synthetic.



The user asks: 'Add this meeting note to Jordan Lee at Acme.' Search returns two Jordan Lees, one at Acme and one at Beta, with the Beta record first. The write tool accepts either ID and returns success.



Counting a successful tool call would miss an update to the wrong contact. I'd check the final state instead: the Acme record has the note, the Beta record is untouched, and the response identifies which contact was updated. Then I'd run a second case with two Jordan Lees at Acme and no other identifying detail; that one should ask for clarification before writing.



For people evaluating agents that edit records, do you test this at the tool-selection level, by checking database changes, or both? I'm especially interested in how you score a correct clarification request without rewarding an agent that asks unnecessary questions on every unambiguous task.

document fetched 2026-10-13 00:20 · canonical https://reddit.com/r/AI_Agents/comments/1x3p11p/how_do_you_test_that_an_agent_updates_the_right