← sources

What do you check after a retrieval migration beyond matching record counts?

source: reddit (reddit:AI_Agents)published 2026-10-25 06:15ingested 2026-10-26 02:35

retained text

Suppose the source and destination both report 10,000 records after a migration. One missing record and one extra record would already make those counts agree.



For a product-search agent, I'd start with a small frozen catalog: two similar items, one out of stock, and another available only in a different region. Then ask for an available item in a specific region. A plausible recommendation isn't enough to show the availability and region filters survived the move.



I'd compare original IDs and payloads first, then the eligible record set for each required filter. After that, I'd run the agent's actual questions and check the answer and cited sources. Those checks can fail independently.



If the ranking method changes, I'd evaluate relevance and answer support rather than require identical result ordering. But a highly relevant item that violates a required filter still fails the task.



I'd also save the source snapshot and query configuration with the comparison so later catalog changes don't get mistaken for migration errors.



Which part of that verification tends to get missed when you move an agent's retrieval backend?

document fetched 2026-10-26 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xeur4g/what_do_you_check_after_a_retrieval_migration