# what do you score when the failure only exists across five turns? — retained record

> Retained reddit record from reddit:AI_Agents.

Canonical: https://highsignal.app/data/records/b4bc0741844d1d8b

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:AI_Agents)  published 2026-10-10 18:23 ingested 2026-10-11 02:36

[original source ↗](https://www.reddit.com/r/AI_Agents/comments/1x2m9tt/what_do_you_score_when_the_failure_only_exists/)

## retained text



```
how do you test a memory failure when every reply passes on its own? across 4 product teams we've got 12800 travel sessions at 5 turns each, exactly 64000 turns, and per turn relevance is 97.6%. clean on paper. but 947 sessions, about 7.4%, reoffer an itinerary that was rejected earlier because compressed conversation state drops the rejection before turn 4 (a reviewer finally realized every individual reply looks reasonable in isolation). the trace group shows the reoffer, the local score never sees it.


so we're trying to decide what belongs in session level scoring versus a behavior spec. multi turn evaluation can check that a rejection survives the whole session, while trajectory scoring can catch the reoffer when it happens. I can make the deterministic state assertion strict but I'm less sure how to score legitimate reconsideration without treating it as the same failure.

what are you using to distinguish a real preference reversal from lost conversation state across several turns?
```



document fetched 2026-10-11 00:18 · canonical https://reddit.com/r/AI_Agents/comments/1x2m9tt/what_do_you_score_when_the_failure_only_exists
