Moonshot's Kimi K3.1 Pro Tops Every Public API Model in Security Benchmarks

retained text

Moonshot's Kimi K3.1 Pro Tops Every Public API Model in Security Benchmarks
Moonshot's new reasoning model takes the top open spot on Artificial Analysis Cyber Index at 64%, trailing only a trusted-access OpenAI model.
- Kimi K3.1 Pro scores 64% on Artificial Analysis Cyber Index, #1 among openly-available models.
- Only trusted-access GPT-6 Sol (Daybreak Blue, max) ranks higher at 69%.
- Beats previous open leader DeepSeek V4.1 Flash 1017 by 7 points, up 23 from Kimi K3.
- Highest CyberGym-E2E-AA score of any model tested at 95%, ahead of GPT-6 Sol.
- No safety refusals, versus heavy refusal rates on proprietary models' public APIs.
- Cost per Task of $2.81, 15% higher than K3 despite 33% cheaper token pricing.
Moonshot’s Kimi K3.1 Pro has earned the highest Cyber Index score among models available through a standard public API. Its 64% result trails the trusted-access GPT-6 Sol configuration by five percentage points and leads the next public option by seven. The result gives developers a strong candidate for repository-level vulnerability detection and repair, subject to uneven benchmark performance, higher task costs, and the current lack of downloadable weights.
Three repository tests, one score
Artificial Analysis combines three evaluations into the Cyber Index, which measures how well models complete defensive security work in real software repositories. Models inspect source code, identify vulnerabilities, reproduce failures, and produce patches without breaking existing behavior. The evaluations exclude requests to construct working exploits.
- CWE-Bench-AA: an Artificial Analysis adaptation of Collinear AI’s CWE-bench.
- DeepsecBench-AA: an adaptation of Vercel’s DeepsecBench.
- CyberGym-E2E-AA: an adaptation of Berkeley RDI’s CyberGym-E2E.
The separate Artificial Analysis Intelligence Index gives K3.1 Pro a score of 55, which is 11 points above Kimi K3. On the Cyber Index, the increase is larger: K3.1 Pro reaches 64%, up 23 percentage points from K3’s 41%.
CyberGym drives the leap
K3.1 Pro’s largest disclosed sub-benchmark improvement appears on CyberGym-E2E-AA, where it rises from Kimi K3’s 58% to 95%. That score leads every tested model, including the trusted-access GPT-6 Sol configuration. The 37% DeepsecBench-AA result and 61% CWE-Bench-AA result show that its advantage is concentrated in end-to-end vulnerability reproduction and repair rather than spread evenly across the suite.
Refusals reshape usable performance
Artificial Analysis recorded no safety refusals from K3.1 Pro during the evaluation. Models from OpenAI, Anthropic, and Google refused some security prompts, lowering their completed-task scores. GPT-6 Sol’s trusted-access “Daybreak Blue, max” configuration scores 32 points above its public counterpart, illustrating how access policies can alter effective performance in an automated security pipeline.
A zero-refusal benchmark run does not guarantee that every production request will be accepted. It does show that K3.1 Pro completed this defensive test suite without policy blocks, an operational advantage for scanners and patching agents that run without continuous human intervention.
Filtering leaves a measurable dent
Moonshot says it removed cyber-related data from both training stages for the planned open-weight release while retaining general-purpose coding tasks. Its internal xdev-bench-hard ablation quantifies the tradeoff: an unfiltered checkpoint solves 90.5% of problems at pass@3, while the final K3.1 Pro reaches 81.0%. Pass@3 counts a problem as solved when at least one of three attempts succeeds.
The 9.5-point internal decline indicates that Moonshot’s risk-reduction measures reduced capability on its own evaluation. That ablation does not establish the cause of any individual Cyber Index result, but K3.1 Pro’s 64% public score shows that substantial defensive capability remained after filtering.
Token use pushes up task cost
Moonshot cut K3.1 Pro’s list price to $2 per million input tokens and $10 per million output tokens, down from K3’s $3 and $15 rates. Artificial Analysis nevertheless records roughly twice the input-token consumption across the Cyber Index tasks. The added usage raises benchmark cost per task by 15%, from $2.45 for K3 to $2.81 for K3.1 Pro.
Where each option fits
- High completion rates: K3.1 Pro offers the strongest overall Cyber Index score through a standard public API and recorded no refusals in the evaluation.
- End-to-end repair: Its 95% CyberGym-E2E-AA result favors workflows centered on failure reproduction and patch generation.
- Cost-sensitive scanning: DeepSeek V4.1 Flash 1017 costs $0.42 per task, about one-seventh of K3.1 Pro’s cost, while trailing by seven index points.
- Maximum benchmark performance: GPT-6 Sol (Daybreak Blue, max) leads by five points and costs less per task, but access is restricted.
- Self-hosted deployment: K3.1 Pro is currently available through Moonshot’s API, with model weights expected in a later release.
Moonshot shipped K3.1 Pro three months after Kimi K3, retaining the same 2.8-trillion-parameter base and adding further training. The benchmark supports its use in defensive coding agents, vulnerability scanners, and patch-generation systems, while leaving offensive security capability and production performance outside the evaluation’s scope.

document fetched 2026-10-25 10:34