loading
loading
DeepSeek's V4.1 Flash 1017 Tops Public Cybersecurity Rankings at $0.42 per Task DeepSeek V4.1 Flash 1017 tops the Artificial Analysis Cyber Index among open models, with zero safety refusals and a 53 point jump on CyberGym-E2E-AA. - DeepSeek V4.1 Flash 1017 scores 57% on the Artificial Analysis Cyber Index, #1 among open models - Beats predecessor by 16 points, driven by 53-point jump on CyberGym-E2E-AA (23% to 76%) - Zero safety refusals, versus heavy refusal rates from proprietary frontier models on security tasks - Costs $0.42 per task, 4x cheaper than leading trusted-access GPT-6 Sol at $1.77 - 552B MoE with 1M context, MIT-licensed, served as deepseek-flash - DeepsecBench-AA score actually regressed from 31% to 25%, so gains are uneven DeepSeek’s refreshed V4.1 Flash 1017 checkpoint scored 57% on the Cyber Index, the highest result among publicly available models evaluated. It finished one point ahead of Grok 4.7 and MiMo-V2.6-Pro, and second overall behind an OpenAI model restricted to trusted-access customers. Three tests, one defensive loop The Cyber Index evaluates three stages of defensive security work: finding vulnerabilities in source code, reproducing and validating them, and producing patches that preserve existing functionality. The suite combines: - CWE-Bench-AA, from Collinear AI - DeepsecBench-AA, from Vercel - CyberGym-E2E-AA, from Berkeley RDI Artificial Analysis calculates the index as an equal-weighted average of CWE-Bench-AA pass@1, the median DeepsecBench-AA F2 score across three runs, and CyberGym-E2E-AA pass@1. Pass@1 records success on a single attempt. F2 combines precision and recall while assigning more weight to recall. Agents inspect source, identify weaknesses, reproduce failures, modify code, and run tests; deployable exploit construction falls outside the evaluation. CyberGym supplied the leap V4.1 Flash 1017 improved 16 points over the previous checkpoint, rising from 41% to 57%. The component results moved unevenly: - CyberGym-E2E-AA: rose 53 points, from 23% to 76% - CWE-Bench-AA: increased slightly to 69% - DeepsecBench-AA: fell six points, from 31% to 25% CyberGym’s increase contributes 17.7 points to the equally weighted aggregate before the changes in the other two tests are applied. The result points to stronger multi-step behavior, including reproducing a bug, editing code, and verifying a patch, while performance across vulnerability classes remains uneven. Public leaderboard and task cost At $0.42 per benchmark task, 1017 lies on the reported capability-cost Pareto frontier, meaning no evaluated model both scores higher and costs less. MiMo-V2.6-Pro costs about 2.3 times less for a score one point lower. GPT-6 Sol costs roughly four times more and adds 12 points. Production spending will vary with prompt length, output length, serving configuration, and workload. DeepSeek’s API charges $0.30 per million input tokens and $1.20 per million output tokens during peak hours, with rates halved off-peak. The checkpoint replaced the previous V4 Flash deployment, so requests to deepseek-v4-flash now route to 1017. Refusals reshape the scores Safety filters materially affect the ranking because vulnerability discovery can resemble attack preparation to a refusal classifier. A refused task receives a zero, which means deployment policy influences the index alongside code-analysis and patching ability. Anthropic, Google, and OpenAI models declined tasks during the evaluation. OpenAI’s trusted-access leader scored 32 points above the provider’s publicly accessible counterpart, with the difference driven almost entirely by fewer refusals. DeepSeek V4.1 Flash 1017 recorded no safety refusals. Deployments therefore need repository permissions, sandboxing, command controls, and review gates suited to security automation. Inside the 1017 checkpoint DeepSeek V4.1 Flash 1017 is an October refresh within the V4.1 Flash family. Its published specifications include: The mixture-of-experts architecture routes each token through a subset of the model’s weights, reducing active computation relative to its 552-billion-parameter total. Hosting still requires substantial storage and serving infrastructure despite the sparse activation. DeepSeek says the checkpoint uses reinforcement learning with self-feedback for end-to-end project work, with examples spanning game creation and 3D modeling. It is tuned for DeepSeek Harness, Tencent CodeBuddy, and WorkBuddy. That agentic post-training offers a plausible explanation for the CyberGym gain because the benchmark rewards repeated reproduction, editing, and testing loops. Where 1017 fits The checkpoint suits security workflows that benefit from open weights, long context, and multi-step code editing: - Repository-scale vulnerability triage that groups CWE candidates and proposes patches - Security agents that reproduce reported crashes and generate minimal test cases - Internal code-review tools that keep sensitive source within a controlled environment - Workflows that encounter refusals from hosted models when processing security-related code Limits behind the 57% DeepsecBench-AA regressed from 31% to 25%, leaving the older V4.1 Flash checkpoint stronger on that part of the evaluation. The trusted-access leader also remains 12 points ahead, and the open model’s size complicates self-hosting even with sparse activation. The 57% result is an aggregate benchmark score rather than an expected vulnerability-detection rate for an arbitrary repository. Teams evaluating 1017 should test it against their own languages, vulnerability classes, build systems, and patch-acceptance criteria, then run generated changes in a sandbox and require normal code review before merging. For organizations that cannot send proprietary source to an external API, 1017 is a credible candidate for an internal security agent. Its strongest evidence supports vulnerability reproduction and patching workflows, while the DeepsecBench decline warrants workload-specific testing before it replaces an existing model.
document fetched 2026-10-24 15:22