← sources

Blind review exposed how much our rubric depended on the first score

source: reddit (reddit:deeplearning)published 2026-10-18 20:35ingested 2026-10-19 02:38

retained text

We had 480 model outputs reviewed by 3 people, which gave us 1440 total reviews. With prior scores and comments visible on the 2nd and 3rd passes, the report counted 1022 matching judgments across those 1440 reviews. We treated that as solid inter rater agreement and decided the rubric was stable. The scores matched and our human scoring looked calibrated. The agreement chart looked very pleased with itself.

So we reran the same set as a blind review. Matches dropped to 749 and 96 rows showed different readings of the rubric. Watching agreement fall as soon as the prior comments disappeared was a rough meeting. Removing score visibility made reviewer anchoring hard to ignore. Our rubric calibration had trained people to align with an existing judgment and hidden where the criteria were ambiguous. The adjudication queue now holds cases where 1 reviewer marks partial compliance as pass while another marks it as fail. I’m trying to decide whether those 96 rows need rewritten criteria, more calibration examples or fresh reviewers before we trust the next agreement number. 



How are you separating a stable rubric from reviews that only converge after someone sees the 1st score

document fetched 2026-10-19 00:23 · canonical https://reddit.com/r/deeplearning/comments/1x9don5/blind_review_exposed_how_much_our_rubric_depended