← sources

I built EvalSeal v2.2.0: reproducibility receipts for LLM and agent evals

source: reddit (reddit:AI_Agents)published 2026-09-28 07:59ingested 2026-09-29 02:36

retained text

I’ve been working on EvalSeal, an open-source tool for making LLM eval results easier to trust.

The problem I kept running into:

A score changed, but I couldn’t always tell whether the model improved, the judge changed, the prompt shifted, or the eval itself was noisy.

EvalSeal now runs eval cases multiple times, measures per-case instability, captures provenance, and seals the result into a tamper-evident ledger.

Current v1.5.0 includes:

* per-case verdict strips
* evaluator fingerprinting
* signed ledger heads
* CI gates
* drift comparison with `evalseal diff`
* deterministic HTML receipts you can attach to a PR
* replayable examples with no API key

One finding from the repo:

Using an LLM judge, 5 of 20 borderline cases flipped verdicts across repeated runs.

Using numeric answer matching on 40 GSM8K cases, 0 cases flipped.

Same model family, different grading method. The instability came from the evaluator, not the target model.

Would love feedback from anyone working on evals, CI gates, promptfoo-style workflows, or agent reliability.

document fetched 2026-09-29 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1ws8ix4/i_built_evalseal_v220_reproducibility_receipts