I built EvalSeal v2.2.0: reproducibility receipts for LLM and agent evals
retained text
I’ve been working on EvalSeal, an open-source tool for making LLM eval results easier to trust. The problem I kept running into: A score changed, but I couldn’t always tell whether the model improved, the judge changed, the prompt shifted, or the eval itself was noisy. EvalSeal now runs eval cases multiple times, measures per-case instability, captures provenance, and seals the result into a tamper-evident ledger. Current v1.5.0 includes: * per-case verdict strips * evaluator fingerprinting * signed ledger heads * CI gates * drift comparison with `evalseal diff` * deterministic HTML receipts you can attach to a PR * replayable examples with no API key One finding from the repo: Using an LLM judge, 5 of 20 borderline cases flipped verdicts across repeated runs. Using numeric answer matching on 40 GSM8K cases, 0 cases flipped. Same model family, different grading method. The instability came from the evaluator, not the target model. Would love feedback from anyone working on evals, CI gates, promptfoo-style workflows, or agent reliability.
document fetched 2026-09-29 00:19 · canonical https://reddit.com/r/AI_Agents/comments/1ws8ix4/i_built_evalseal_v220_reproducibility_receipts