Do your agent evals test access being revoked halfway through a conversation?
retained text
One eval case I'd add for an agent working with permissioned data: revoke access halfway through the conversation. Toy setup: - A user can access Project A. The agent retrieves an incident note and produces a summary. - Their project access is removed. - In the same session, they ask for another detail from that note. If the application's policy requires current access, what should happen? A fresh retrieval might be blocked while the old snippet, a memory entry or a cached answer is still available to the model. I'd test those paths separately: 1. A follow-up in the existing conversation. 2. The same question from another account through a shared cache. 3. A run that starts before revocation but produces its answer afterward. For each case, I'd record the applicable access policy, the evidence supplied and when authorization was checked. A refusal on a new search wouldn't establish that the existing session or cache also respects the change. This is about preventing a later disclosure under that policy. It can't take back information the user already received. Where would you enforce this in your stack: retrieval, context construction, the response boundary, or a combination?
document fetched 2026-10-21 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1xam18t/do_your_agent_evals_test_access_being_revoked