← sources

Our helpfulness judge blocked a PR that QA thought was fine

source: reddit (reddit:MLOps)published 2026-11-01 20:20ingested 2026-11-02 02:39

retained text

A PR for our customer-response assistant got held because a model-graded helpfulness score dropped on an address-change reply. The order had already shipped, and our QA reviewer thought the short "we can't change it now, here's how to contact the carrier" response was appropriate.



The fixture checks passed: shipped status was reflected correctly, the required handoff was present, and no address-change action was attempted.



I reran the saved cases with the same scorer rubric in Braintrust and inspected the disputed result. The judge seemed to want a longer, more sympathetic response. Fair feedback maybe, but we hadn't calibrated that threshold well enough with reviewers to justify blocking a merge.



We changed the CI rule. Contract and integration checks still block. A drop in model-graded helpfulness now creates a review item with the case and rubric but doesn't fail the PR on its own. A reviewer can still hold the merge.



Anyone made a similar split or found a defensible way to let subjective scores block?

document fetched 2026-11-02 00:26 · canonical https://reddit.com/r/mlops/comments/1xldbud/our_helpfulness_judge_blocked_a_pr_that_qa