How do you score "insufficient visual evidence" in multimodal evals?
retained text
Working on a photo based bicycle repair triage prototype and a mechanic reviewing sample outputs pointed out a flaw in our evaluation criteria. Several photos don't show enough detail to assess component wear reliably. The model will still name a plausible defect and our current rubric tends to reward that over saying another photo or physical inspection is needed. I've been running image and description cases through Braintrust evaluations and inspecting the individual judgments. Some responses that received favorable scores were ones the mechanic said he couldn't responsibly make from the evidence shown. Considering adding an assessability label to each test case then scoring diagnostic accuracy separately from how the model handles missing visual evidence. I want to avoid a rubric where asking for another photo becomes an easy way to score well, even when the original image is adequate. How are you handling cases where the image doesn't show enough detail? Are you scoring those separately or letting the same rubric account for missing evidence?
document fetched 2026-11-18 00:28 · canonical https://reddit.com/r/mlops/comments/1xykfu0/how_do_you_score_insufficient_visual_evidence_in