# I built EvalSeal v2.2.0: reproducibility receipts for LLM and agent evals — retained record

> Retained reddit record from reddit:AI_Agents.

Canonical: https://highsignal.app/data/records/8fbcc8a18a7c65a9

[← sources](https://highsignal.app/data)


source: [reddit](https://highsignal.app/data/reddit) (reddit:AI_Agents)  published 2026-09-28 07:59 ingested 2026-09-29 02:36

[original source ↗](https://www.reddit.com/r/AI_Agents/comments/1ws8ix4/i_built_evalseal_v220_reproducibility_receipts/)

## retained text



```
I’ve been working on EvalSeal, an open-source tool for making LLM eval results easier to trust.

The problem I kept running into:

A score changed, but I couldn’t always tell whether the model improved, the judge changed, the prompt shifted, or the eval itself was noisy.

EvalSeal now runs eval cases multiple times, measures per-case instability, captures provenance, and seals the result into a tamper-evident ledger.

Current v1.5.0 includes:

* per-case verdict strips
* evaluator fingerprinting
* signed ledger heads
* CI gates
* drift comparison with `evalseal diff`
* deterministic HTML receipts you can attach to a PR
* replayable examples with no API key

One finding from the repo:

Using an LLM judge, 5 of 20 borderline cases flipped verdicts across repeated runs.

Using numeric answer matching on 40 GSM8K cases, 0 cases flipped.

Same model family, different grading method. The instability came from the evaluator, not the target model.

Would love feedback from anyone working on evals, CI gates, promptfoo-style workflows, or agent reliability.
```



document fetched 2026-09-29 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1ws8ix4/i_built_evalseal_v220_reproducibility_receipts
