Instead of arguing about whether Claude has been “nerfed,” why don't we measure it?
retained text
'Nerfing' could be a strategy to maximize uptime and revenue during periods when demand from paying customers exceeds the compute available for inference. Whatever the cause, the unpredictable variability in performance from week to week creates a frustrating lack of consistency in output quality. Has anyone considered building a web app that could objectively test whether a model has been 'nerfed'? The idea would be for the app to maintain a large, secret collection of coding prompts, randomly sampling a small subset each time someone runs a test. You give those prompts to your model and submit its outputs to the app. The app then analyzes the outputs and estimates which model (Opus 5.5, Opus 5, Sonnet 5.5, Sonnet 5, etc.), or combination of models, was most likely to have produced them. Alternatively, the app could establish a fingerprint for Opus 5.5 when it's performing normally and then test whether that fingerprint changes over time, or differs across users at any given moment in time. This might be easier than trying to predict exactly which model (or combination of models) the output resembles, since nerfing may involve quantization or other compute-saving tactics that lead it to no longer resemble any of the previous un-nerfed frontier model releases. Model fingerprinting from input/output behavior is already an area of research. Is anyone aware of a project that's doing this as a service for users, specifically to track changes in the behavior of commercial models over time?
document fetched 2026-10-03 00:23 · canonical https://reddit.com/r/ClaudeAI/comments/1wvhknf/instead_of_arguing_about_whether_claude_has_been