loading
loading
Artificial Analysis' AA-Omniscience v1.1 Catches AI Models Guessing Instead of Abstaining Artificial Analysis refreshed its knowledge and hallucination benchmark, cutting it to 3,000 questions and exposing which models were quietly overfitting. - AA-Omniscience v1.1 shrinks to 3,000 questions across 6 domains, removing saturated and ambiguous items. - Gemini 4 Carbon leads with Net Score 54, ahead of Claude Opus 5.5 (48) and Claude Fable 5.1 (46). - Rank order holds with 0.98 Spearman correlation to v1, but median Net Score drops 0.6 points. - MiniMax M3.1 Flash collapses 22 points to -6; MiMo V2.6 Pro loses 13, suggesting contamination. - Claude Fable 5.1 tops accuracy at 68%; Gemini 4 Argon has lowest hallucination rate at 4%. - New public 300-question set on Hugging Face correlates 0.99 with the private benchmark. Artificial Analysis has released AA-Omniscience v1.1, an updated benchmark for factual knowledge and calibration. It measures whether a model answers correctly, declines when uncertain, or invents an answer. The leading models retained similar positions, while several individual scores fell sharply after the question set changed. A smaller test with sharper questions Version 1.1 contains 3,000 private questions across six domains, down from the 6,000 questions described for v1. Artificial Analysis removed questions that most models had mastered, revised ambiguous items, and added new material. Artificial Analysis also released a recalibrated 300-question public set. Across evaluated models, its Net Scores have Pearson and Spearman correlations of 0.99 with the private set, while the median public-private difference is 0.0 points. Pearson correlation measures agreement between scores; Spearman correlation measures whether the model rankings stay aligned. Scoring rewards restraint Each question contributes to the AA-Omniscience Index according to a simple rule: - Correct answer: +1 - Incorrect answer: -1 - Abstention: 0 The normalized Net Score ranges from -100 to 100 and can be expressed as accuracy − hallucination rate. A score of 0 means the model produced equal numbers of correct and incorrect answers. Negative scores indicate more incorrect answers than correct ones. A model that answers 40% of questions correctly, answers 15% incorrectly, and abstains on the remaining 45% receives a Net Score of 25. This structure gives models credit for useful knowledge while penalizing unsupported guesses. Frontier models keep the lead Muse Spark Open leads the open-weights entries and trails the overall leader by roughly 28 points. The result establishes a gap on this benchmark; coding ability, reasoning, latency, cost, and deployment flexibility require separate evaluation. Configuration labels such as High, Max, Xhigh, and Default Fallback identify distinct leaderboard entries. Reproductions and vendor comparisons should preserve the exact model version, inference configuration, system prompt, and decoding settings. The refresh moves four models sharply Across evaluated models, the average Net Score declined by 1.9 points and the median declined by 0.6 points. Most rankings remained stable, but four reported changes were substantially larger. MiniMax M3.1 Flash’s move from +16 to -6 crosses a meaningful scoring boundary: under v1.1, it produced more incorrect answers than correct ones after abstentions were excluded from the penalty. Large drops raise contamination questions An overall rank correlation of 0.98 means the ordering remained similar across releases. Large changes for a small number of models are consistent with exposure to old public questions during training, especially when benchmark items have circulated widely. Score changes alone cannot establish contamination. Revised wording, domain composition, grading behavior, model updates, sampling settings, and sensitivity to particular questions can also produce large deltas. A controlled contamination audit would require training-data evidence or experiments that compare exposed and unexposed items. The close alignment between the new public and private sets reduces the need to expose private questions. Its reported calibration applies across the models tested by Artificial Analysis, while individual models may still diverge because a 300-question sample cannot capture every behavior. Choose metrics by failure cost Benchmarks that require an answer to every question can reward guessing. AA-Omniscience instead exposes the trade-off among accuracy, hallucination, and abstention, which matters when an unsupported answer carries more risk than a refusal. The median hallucination rate fell by 1.4 percentage points on v1.1. Because the questions changed, that decline should be interpreted as a property of performance on the refreshed set rather than a general improvement in model honesty. AA-Omniscience covers parametric knowledge under benchmark prompts. Retrieval quality, citation accuracy, tool use, long-context behavior, prompt-injection resistance, and domain-specific workflows remain outside its scope. Teams can use the benchmark as an initial screening signal and follow it with evaluations built from representative production tasks. Run the public set carefully The 300-question public set is available on Hugging Face, while the underlying research paper describes question generation and grading. - Pin the model version, provider, system prompt, inference configuration, and decoding settings. - Run all 300 questions using the published prompts and grading method. - Report Net Score, accuracy, hallucination rate, abstention frequency, and domain-level results. - Repeat a sample when outputs are stochastic and record the resulting variance. - Follow the benchmark with a private evaluation drawn from the intended application. The reported 0.99 correlations and median 0.0-point difference support using the public set as a preliminary estimate of private leaderboard performance. Exact results can still vary with model-specific behavior, serving infrastructure, and evaluation settings.
document fetched 2026-10-31 02:55