fine tune went from 61% to 94% on our benchmark, then we found 300 of the 1000 items were in the training set. what does the score measure now?
retained text
chose b. not confident i went with b because 300 out of 1000 is not the whole benchmark, so 700 clean items should still be carrying most of the score. if the model got better on those too, the gain is real. but i cant tell from one number whether the jump came from the 700 or the 300, and that feels like the whole question
document fetched 2026-09-27 00:24 · canonical https://reddit.com/r/deeplearning/comments/1wqyuws/fine_tune_went_from_61_to_94_on_our_benchmark