Mistral’s new Large 4 trails some Chinese open models in independent tests

retained text

Independent tests rank Mistral’s new trillion-parameter Large 4 the best AI model outside the U.S. and China — but Chinese open-weights still overcome Europe's best efforts
Its best result is a cyber test, but it falls behind on the index.
Mistral launched its trillion-parameter Large 4 (ML4) model this week, saying it “pushes the frontier of open-weight performance.” Artificial Analysis (AA), an independent AI benchmarking firm, ranked Mistral's Large 4 as the best model available from outside the U.S. and China in its index. The mixture-of-experts model, trained on Nvidia’s Grace Blackwell GPUs, scored its personal best in AA’s CyberGym-E2E-AA end-to-end cybersecurity measure, although several Chinese open models beat it overall.
ML4 scored 38 on the Artificial Analysis Intelligence Index v4.3.2, which AA says makes it the “most intelligent model from outside the US and China." At least five Chinese open models scored higher on the index: DeepSeek V4.1 Flash (max) at 39, GLM-5.3-Flash at 42, Kimi K3 (max) at 44, GLM-5.3 (max) at 45, and MiMo-V2.6-Pro at 46. While MiMo-V2.6-Pro is comparable in total size to ML4, GLM-5.3-Flash is about one-third the size at 320B parameters total and 18B versus ML4’s 49B–52B active. Unlike the rest, ML4 isn’t open yet: Mistral says its weights are due “by the end of the month.”
Cost is a different story. ML4 comes in at $1.13 per AA task at list price, or $0.57 at the 50% discounted launch price. This compares to $0.13 for MiMo-V2.6-Pro and $0.25 for GLM-5.3-Flash. Although comparatively expensive, ML4 scored what AA calls its “strongest result” and Mistral calls “the highest of any model” with an 81.7% score on CyberGym-E2E-AA, compared with GLM-5.3-Flash’s 74%, from a model whose post-training run Mistral claims is “still in flight.”
While the score is impressive, it is only one of three tests in the Cyber Index; ML4 only scores 16% in DeepsecBench-AA and 51% in CWE-Bench-AA. However, AA’s projection is that, once the weights ship, it “will rank among the top three open weights models on the Cyber Index.” This leaves the door open for ML4 to apply to narrower tasks.
Mistral bets on this, saying ML4 is “state-of-the-art among open models” on enterprise workloads such as “cybersecurity, finance and law,” and that in visual grounding, where its 1.6B-parameter vision encoder comes in, it can go as far as “surpassing even frontier closed models.” On AA’s Cyber Index, Mistral says, ML4 ranks “among the top five models globally,” and it solves 93% of Cybench, a set of 40 challenges drawn from security competitions.
CEO Arthur Mensch went further in Abu Dhabi, saying the model is “above the Chinese models on certain aspects, including cyber,” Reuters reported. Mistral’s other figures include DeepSWE v1.1 at 61.7%, Coding Agent Index at 49.8%, AutomationBench at 59.9%, and Dense 200 at 42%; AA’s own chart puts ML4 at 27% on Terminal-Bench 4.0.
ML4 does, in fact, beat OpenAI’s closed GPT-6 Luna (max) on CyberGym-E2E-AA, but Luna matches ML4’s 38 on AA’s index at about one-sixteenth of the cost per task ($0.07). Mistral says frontier models such as Claude Opus 5.5 and GPT-6 Astra “score near zero on the same test because they refuse to perform the task,” and AA’s chart marks both as declining some tasks on safety grounds.
Outside of that test, Mistral says ML4 also beats DeepSeek V4 Pro 0813 and Qwen3.8 Max on the Coding Agent Index; per AA, it is cheaper than the latter. Overall, AA’s index-vs.-cost chart puts ML4 below the Pareto line and just outside the “Most attractive quadrant.”
The cost comes from needing 200 million output tokens to run AA’s index, against a median of 81 million. As for speed, ML4 generated 116.1 tokens per second, with 1.46 seconds to the first token. This puts ML4 at about 33% faster than its tier median and 2.6x faster to the first token. AA’s speed is measured on Mistral’s own API, which Mistral says runs on the same hardware the model used for training.
ML4 was “trained from scratch on 3,800 NVIDIA Grace Blackwell GPUs in Mistral’s own datacenters in Europe,” the company said. This is up from 3,000 Nvidia H200s used to train Large 3, released last year, which was smaller at 675B total parameters with 41B active. The training occurred over “roughly two months,” as reported by VentureBeat.
Mistral raised a €3 billion Series D in what the company calls “the largest equity round ever raised by a European technology company” in its post. Mistral Compute has planned for 18,000 Grace Blackwell chips, as announced with Nvidia last year. Mistral says it is “significantly scaling up our compute capacity in our own European datacenters,” and that ML4 will become the base for “a new generation of specialized and optimized Mistral models.” The model’s open weights, once released, will be able to run on a private cloud or on-premises.
The weights, architecture details, and additional benchmarks are expected before the end of October. Artificial Analysis lists the preview as proprietary until the weights arrive. Once it does, the model may be re-scored, and Mistral’s claims will be held to task. For the first two weeks following release, ML4 can be used via API at a discounted rate.
While European models still have a lot to prove, based on previous performance and ML4’s showing with an 8-point gap versus a similar-size Chinese model that’s also a lot less expensive, Mistral has demonstrated resolve and produced a model with, at least, niche application, as the basis to eventually close that gap.
Shane Downing is a Contributing Writer for Tom’s Hardware, covering consumer storage, PC hardware, and AI.

document fetched 2026-10-10 02:48 · canonical https://tomshardware.com/tech-industry/artificial-intelligence/independent-tests-rank-mistrals-new-trillion-parameter-large-4-the-best-ai-model-outside-the-u-s-and-china-but-chinese-open-weights-still-overcome-europes-best-efforts