loading
loading
We — Nathan and Florian — wanted to discuss all the messy trade-offs of a busy week in AI and ask all the pressing questions like: - Why did Kimi K3.1 Pro have a mixed reaction with such incredible benchmarks? Will it be adopted? - Were we wrong about the risks of open-weight models and cybersecurity? - Why do cyber experts discuss recent events so differently than AI experts? - What comes next in the few months? What will change our minds? We will write more about this subject, but wanted to bring you behind the curtain as we aspire to quickly learn more about the cybersecurity industry and trends across the last few years that have been leading us today. We conclude the podcast with some lighter topics, from Muse-maxing to excitement in still hiring junior technical talent. Please send us questions and topics, as we plan to record more of these podcasts going forward. Listen on Apple Podcasts, Spotify, and wherever you get your podcasts. For other Interconnects podcasts, go here. YouTube ↓ Chapters - 00:00 Kimi K3.1 Reaction - 08:15 Kimi’s open-weight cyber principles - 11:23 You can hack everything era - 15:54 PHP2Shell, LiteLLM, and the new internet - 24:00 Abliterated APIs, safeguards, and trusted access - 31:41 Anthropic not hiring juniors employees - 36:16 Florian’s Muse addiction - 45:30 Wrap-up Links we discuss - My thread with Dean Ball about the “boy who cried wolf” story of the frontier labs - Kimi K3.1 Pro announcement blog - Kimi K3.1 Pro on Artificial Analysis Intelligence Index - Kimi K3.1 on Artificial Analysis defensive cyber benchmarks - Program Bench - Open-weight Mythos would’ve been fine take - PHP “hasn’t caused a disaster yet” update - Abliteration.ai serving abliterated models API - Morph serving abliterated DeepSeek via API - Joshua Saxe’s Substack on AI and cybersecurity - Azeem Azhar, “The internet’s Wuhan moment” - Anthropic, “How employment is changing at Anthropic” - LiteLLM security incident report Transcript 00:00:00 Nathan Lambert: We are back with another Interconnects podcast. We are with Florian, eval guy, recently moved to SF. Florian, is Kimi K3.1 Pro, name is a mouthful, going to live up to the benchmark scores, for example on Artificial Analysis, or where is it going to land? 00:00:17 Florian Brand: In the past, the Kimi models always lived up to the expectations, but this release is kind of weird compared to the others. I think it is also partially because they are becoming more professional, and the previous model drops were kind of out of nowhere. This time, which also got some criticism, they launched a bigger social media campaign, which allegedly also involved paying people to post about the model without disclosing. 00:00:56 Nathan Lambert: I think a lot of companies do this and tell you not to mention it, but it’s a mess. It’s bad form. No company should do it. Essentially, one person tweeted a screenshot of the press instructions for Kimi that said, “Don’t say this is a paid promotion.” And to my knowledge, no one has corroborated it after that. 00:01:15 Florian Brand: Yeah, and there are right now a few of these model launches, including by American companies, that are just very obviously astroturfed. We have seen launches where it’s thousands of retweets and a few hundred likes, because it works so well to get the attention. Some of them disclose that it is a paid ad while others don’t. But in general, we have seen the professionalization of model launches over time, and pretty quickly. A year ago, everyone just dropped the model, then patched vLLM afterwards, so no one was actually able to run any of the open models for a while. Now you get all the inference frameworks on day one, you get all the inference providers on day one. Or, which is also what we see more and more, the model creator has a few weeks of exclusive access before they release the weights. And that’s also what’s happening with K3.1 Pro. I think it’s two weeks of exclusive access before they release the model weights. 00:02:38 Nathan Lambert: They said the weights would come out by November 4th. And the headline things... I don’t know if I’m going to be able to share screen. I don’t have this set up on this computer. But the headline is this Artificial Analysis number where it’s ahead of GPT-6 Astra. And then if you look at the Kimi K3.1 blog, they have their headline numbers, which are really strong, and then they also have some numbers later on which seem a bit more jagged. I think it was Terminal-Bench Science, which is 20 points behind GPT-6 Astra. I think there were a few others, like ProgramBench. Do you know Terminal-Bench Science and ProgramBench? Are any of these things that people think are important benchmarks for frontier models? Or how do you explain the fact that on Artificial Analysis the model is top, what, three, four? It’s the two latest Claude 5.5 models, Muse Wonder Full, and then Kimi K3.1, which looks crazy. It just looks like totally crazy scores, and I think everybody is still surprised by it. But then there’s this long tail of details that you use to evaluate a model now. I had posted a thread on Twitter of, “What are people’s impressions?” And there’s a lot of people that are like, “This seems not great,” and then some people that are like, “Yeah, it’s fine.” I don’t know, there’s a mixed bag. Relative to K3, which I think was pretty unanimously, this feels like a big model and the post-training seemed a little bit undercooked for that. Maybe this is just a tiny bit RL fried. We’ll see where we land. 00:04:26 Florian Brand: For the benchmarks, one of the things is that the benchmarks that are right now used by Artificial Analysis, they will post their new version probably very soon after we release this podcast. 00:04:43 Nathan Lambert: We can’t tell you the exact date, but both of us know. 00:04:48 Florian Brand: In general, the Artificial Analysis benchmarks are rather short in terms of time horizon, whereas something like ProgramBench is really long and also very explorative. Other benchmarks are telling the model, “All right, this is the goal, and I want you to reach it,” whereas ProgramBench is more like, “All right, here’s what I want, but you need to explore a lot.” And I think that might be the reason that we see some differences: their RL training is still more towards the exploitative side of model training versus the explorative side, where you need to find the solution somehow before you actually know what you’re doing. 00:05:41 Nathan Lambert: But if you’re using a three trillion parameter model, I feel like you kind of want it to be pretty robust at exploration. Because if you’re going to want to use a model for pretty known coding tasks, like using DeepSeek-V4.1-Flash-1017, which is not that far behind on these scores, it’s just going to be way faster. It might not be wildly different on cost per task with the actual token numbers. That’s why I think it’s pretty reasonable criticism of this Kimi release: a big model has got to be... Because it’s going to be slower, it’s got to be like Fable, which is just... I still think Fable is really nice if you’re doing very blurry things. I revert to it when checking my slides over sometimes, where I just feel like it kind of gets me a bit better, where Opus is really, really grindy. I think the open-weight models are better in this kind of grindy perspective, because you can just get them to be faster and cheaper. 00:06:44 Nathan Lambert: It seems like Kimi is gearing up for this Flash-style release with their naming change. Previous Kimi models were just K2, K2.6, K3, and now this is K3.1 Pro. So a lot of the company forecasting, investment-y side of interest on Twitter was, a lot hinges on this for Kimi, because it’s kind of true. We’re not the inference experts, but serving the model seems like you’ve got to commit way more resources to serve this three trillion versus this 700 billion parameter model. You’re going to need NVL72 racks, I’m guessing, to do it well. 00:07:21 Florian Brand: To fit the model, you need B300s, one node per model replica, which is kind of awkward, versus other models which you can even fit on H100s, or H200s rather, which are more readily available. Splitting models is always awkward compared to having one model replica per node, or using existing nodes to fit the full weights. So in that regard, three trillion is just out of, or just in the range of what you can host somewhat reasonably, but it is kind of annoying compared to all the Flash models and all the smaller models, where you can use so many of the existing GPUs that people have. 00:08:15 Nathan Lambert: Do you think Kimi’s shooting itself in the foot a little bit by saying it’s slowing its own cyber capabilities? In the blog post, I can read some of this, because it’s really good for people to say. “Open-weight principles. These evaluations frame our open-weight release decision. First, we aim not to release open weights that would extend the absolute frontier of cyber capability. K3.1 Pro’s offensive cyber capability is, by design, below what our pipeline can produce, and remains behind that of frontier proprietary models.” And they go on to say... I don’t know if the mid-training part is in this, but essentially they manipulated the mid-training data to down-weight cyber capabilities and see that that can last into the post-training. Do you think this would reflect upon the other benchmarks that they have, and maybe some of this long-horizon weakness? Or do you think that people could actually separate this? 00:09:08 Florian Brand: I think people can very well separate these things. I talked to a bunch of other people, and I think that style of release is really good. You want very broad access for everyone to find all the bugs and to patch all the bugs, which is net positive for the world. And Kimi saying they still release the capable model for the defensive side, but they don’t want to advance the offensive side. What I’ve heard from cybersecurity professionals is that exploiting and being able to go through or wreck a whole organization is way harder, because you need very long-horizon capabilities, and you need to persist through all the stages that you penetrate. That’s where closed models, especially GPT-5.6 Sol, a model that’s named often in this regard, are just way, way better than open models. And I think that is important. We want all our code to be secure, and that should happen pretty fast. But for this, you don’t need long horizon. You don’t need all the CTF-style challenges. So throwing these out of the training mix is a reasonable decision, and it helps them position against the fearmongering that once an open model with Mythos-level capabilities, which we arguably have in some regards, is released, the world will end. Which we haven’t seen so far, and I don’t think will happen in the near future. So open models get way more scrutiny than closed models in this regard, and them positioning themselves as, “We are the responsible actors,” is a smart move. 00:11:23 Nathan Lambert: Okay, we can transition into the “you can hack everything” part of this segment. There’s a lot going on here, and Florian and I are both not cyber experts, but it is worth discussing, because I’m kind of of the opinion that the cyber experts that I read talk about this from an ecosystem perspective. All measurements like vulnerability detection, time to exploitation, have been on these exponentials before ChatGPT came out, and there wasn’t a step change as AI models came out, and they focus on this ecosystem perspective. But when you come to the AI people, which is our corner of Twitter, people love to debate open versus closed models and Mythos and things like this. And then the messy middle is, what is the effect of the Daybreak programs and Mythos and things like this on broader society? I feel like I’m trying to muddle through this without getting too stuck into any one point of view. But I really don’t think it is that much of an open versus closed and a capability model thing. 00:12:33 Nathan Lambert: I wish I could go relive the Mythos moment, because my impression was that they made it quickly seem like a cyber weapon. I can go revisit their communications about what they actually said, because it might have been more that they were like, “This is a step change in cyber,” and then the government was very scared of it and took it very, very seriously, and then that pipeline spiraled into this communications of the AI safety community potentially forecasting more problems. It’s hard to hold people accountable for what the media ecosystem does, but also if you are the most prominent company in the world, you are partially responsible for how all of this works. So I’ve gotten pushback from people in the program that are like, “We didn’t say that the world would end when Mythos came out. We just said that there’d be a ton of problems.” And in some ways they are right that there are a ton of vulnerabilities being found. But I do distinctly remember the boy-who-cried-wolf vibe of a lot of the AI safety companies over the last few years. I had this exchange with Dean Ball where I was trying to point this out to him on Twitter, and I was like, “Dog, we’ve been doing this for years.” The boy who cried wolf is a classic parable for a reason, and it seems like a bit of a blind spot, and I just can’t understand it. 00:13:50 Nathan Lambert: But we’re in it, and the vulnerabilities are being found en masse, and I would say there’s a slight uptick in visible hackings. It might be that there’s more hackings within the Twitter community. A lot of cyber people I’ve talked to use reported cyber damages, which a lot of them are through ransomware, as the trend to look at. Historically, AI has been attributed to a very, very low percentage of cyber damages. So if cyber damages do skyrocket with attribution to AI, that is a very difficult world. But the vulnerability finding is very, very unsettling, but we don’t know exactly what it’ll translate into. That’s where I stand, and the reason as to why that translation might not happen. For example, I like Joshua Saxe’s work a lot, and he talks about how the supply chain of cyber hackers is very weak. These people tend to have competition. The population are people that self-select because they cannot get better jobs, and this population of people might also not be people that are willing to do cyber attacks that I would describe as loud and very high damage. So I think there’s a very high likelihood that nation-state actors are using AI to collect information and infiltrate at a large scale. But it’s not clear who the people are in the world that are willing to use AI models to cause mass damage in the physical world. And I don’t like that being... That’s a horrible thing to be like, “Okay, we’re going to figure out whether or not that’s true in the next four months,” but it literally might be where we’re at with the number of vulnerabilities out there. And I think you’ve been following the PHP issue, which I don’t know if you want to summarize for people, and then also some of these things that you’ve been following in the commentary after that. 00:15:54 Florian Brand: So the PHP issue is basically that for years there was a bug which allows you, in PHP, to execute code remotely. So anyone can use any PHP website, which are a lot of the websites on the web. WordPress is one of the biggest ecosystems which runs on PHP. And the OpenAI Patch the Planet program, together with some other AI security researchers or cybersecurity researchers, found this bug and then reported it to the PHP Foundation, worked with other actors to patch this, and then released the patch all at once. Because PHP powers so many parts of the web, and it is a really, really strong exploit, really high risk, a lot of websites even went offline for a while just to patch this, which is the right response. And obviously because it was so high profile, and people think about Mythos and think about open model releases, that has kicked off a number of takes that now everyone who can run an open model basically can hack the world. But the interesting thing is that there are numerous reports that the mass exploitation hasn’t happened and isn’t happening yet. 00:17:37 Nathan Lambert: Wait, did you see the tweet about the DeepSeek swarm fixing it? 00:17:40 Florian Brand: Yeah. 00:17:42 Nathan Lambert: I don’t know if it’s real, but somebody that works at a big cloud company was like, “We’re literally seeing mass fixing at scale by somebody, or a DeepSeek swarm.” Great meme. Coin flip on if it’s real. 00:17:59 Florian Brand: But in general, the expectation of some people seems to be that the second it drops, everyone is able to write code and exploit and is able to scan all the websites and do mass damage, and that doesn’t seem to have happened, which is great for the current state of the world. And I think we will see similar things in the future, so very quick fixes by responsible actors, which is the vast majority. Like you have said, the people who work in cybersecurity are experts at their job. They get those jobs, and there is so much money in fixing things. All the cybersecurity firms, Anthropic and OpenAI, all the AI players as well make a lot of money with these programs, so single nefarious actors are the minority. And nation states, I would say, are completely out of that question, because if you are a nation state, you have completely different capabilities. You might run completely different models and abuse these things for a long time. So I wouldn’t say it’s fair to say that now that open models are becoming better at these things, nation states didn’t have these capabilities for a while, or can’t train models to advance their own capabilities. We are basically seeing that the overall dynamics are staying the same, but are accelerated by a lot. 00:19:49 Florian Brand: The other interesting thing, especially if you think about things like ransomware, is that people still haven’t woken up that they need to patch their systems. I’ve had a few friends who work at very small or medium businesses, and they are like, “Oh, yeah, that sounds horrible. Let me check my PHP version.” And it is so far out of life that they don’t get the security patches, because they didn’t update their PHP versions or their systems for years. So they are vulnerable to a bunch of other security bugs, which still weren’t exploited, which is also very interesting. Another thing that plays into the argument that right now the amount of nefarious actors is very small, and things can stay accessible for way longer than people think, is the LiteLLM hack. LiteLLM is a software library where you can use one endpoint and then ping every provider, which basically allows you to save a lot of lines of code. And they got compromised again this week, which, with all the PHP stuff and other security vulnerabilities, just contributed to the impression that everything is insecure. They did a preliminary report that the reason why they got hacked again is that they had an API key committed to GitHub for five months. So it was accessible in some way, or it was readable for five months, and five months later, someone exploited that and used that to spread malware. 00:21:44 Nathan Lambert: I don’t think we’re in a world where we’re defense dominant, but I don’t think that offensive capabilities being higher is necessarily a lot of the bottleneck, which is very messy. But that’s probably why it’s not changing that much, because yes, the cost and ability to do cyberattacks is going way up, but the institutions that are doing them, that doesn’t necessarily 10X their output in what they’re doing. But the defensive side is worrisome. Azeem Azhar had a good piece on Exponential View, which was just, this is day zero of the new internet, and potentially things will never be as secure and open again, because people are writing exponentially more code with coding agents. There’s just so much more code going out there, and I don’t think flaws and vulnerabilities work in a way where the model writing the code can prevent all the vulnerabilities, because it’s a lot of interactions between software and how you stack these things on top of each other, and I think you need to give the models more context they won’t always have. So this is people’s view, where there’s going to be way more vulnerabilities going forwards, and this transient period of a few years is where we have the backlog. We have the problem of the backlog and exponentially more code being written, so it does seem pretty likely that there’s just going to be way more vulnerability for the next few years, which is not good. 00:23:17 Nathan Lambert: But none of this seems like an open-closed model story. How do you bisect the timing and the forecasting of these things with the fact that open models are going to keep getting better, and that closed models are going to keep getting better while having imperfect safeguards? I kind of feel like the communities actually aren’t doing a good job of saying how the other one informs them. Some people ignore the cyber community, and some people in the cyber community will be like, “This is on trend.” I still don’t know how to fully parse it, because it’s all so overwhelming. 00:23:54 Florian Brand: I would broadly agree with that take. 00:24:00 Nathan Lambert: I’ve been getting a lot of pushback for my open-weight Mythos in April would have been fine take, which was essentially... This is a few weeks ago when I posted this, buried at the end of a cyber article, and at that time there was so much fearmongering that open-weight models were going to cause all these problems and cause all these cyberattacks. A lot of people are essentially saying, “Look, we were right. There’s all these cyber exploits due to open-weight models.” But causality is so hard. I think it would be good to talk to people that are pals with big tech company security leadership and understand, are companies actually deploying open models for defense? I don’t think there are that many companies that are going to be proactive on cyber defense. I get the sense that it’s a pretty reactive industry. So I don’t know. But at the same time, there are these API companies that are serving abliterated models over the API, which seems like just asking to get sued by the government. I don’t know why you would do this. I think the abliterated models for defense is kind of overblown, but it is very real that people want to have models on their own infrastructure so they don’t need to send anything over an API. So to have an abliterated model that you need to send information to over an API in order to secure your systems... I think one of the companies is literally Abliteration.ai, and the other one is Morph. I think it is a bad judgment call for these companies to host these models at this time. 00:25:43 Florian Brand: I also think it remains to be seen how much the abliteration really helps versus it just being very effective marketing. I think the whole thing has shown one thing, and that might play into your open-source Mythos would have been fine take, which is that the diffusion isn’t happening fast enough. People are struggling to get access to, and especially back then were struggling to get access to, the cyber programs from Anthropic and from OpenAI. So they had to use open models. It is obvious that the closed models are just better at these things, but if you can’t get access and get blocked with every request, what should you do? You need to use the less capable open models. And it seems like that has now changed for Anthropic as well. Getting access to the defensive cybersecurity program from OpenAI is rather easy. You submit your ID, you have to secure a system with hardware keys, and then you basically get it, just for the defensive side, for securing your code. And I think that’s amazing. If someone was to abuse it, OpenAI can just look up who the real person behind that is and, in the worst case, go after them or talk to law enforcement. The offensive program is still very hard to get into. And from what I’ve heard, Anthropic is also broadening the access now to more and more people. That’s just positive, and the thing that will change the landscape, because more security researchers and more people being able to sift through so much software and the whole backlog to secure our code is the thing that is needed in the end. 00:27:49 Nathan Lambert: I do think that these programs are a big reason why we’re seeing so many vulnerability disclosures. The PHP disclosure was in the works, and people were in the process of fixing it when it got leaked. I would put more likelihood on people using these closed models easily with trusted access programs to do a lot of those, and I think that companies deserve credit for this. The open-weight model stuff is just really messy. We’re only going to see it when there’s a disaster, and it’s really hard to see usage. I still do think if there was a DeepSeek swarm that hacked an American company, the political fallout would be insane. Maybe some of the politics of this are just going to be slower due to the midterms coming up, where everybody’s focusing on midterms and they’re not focusing on AI right now. I wanted to revisit my open-weight Mythos would be fine piece, and I think the thing that I didn’t do a good job of articulating in it is just what the trend lines forecast in terms of vulnerabilities accelerating, because those trend lines existed. I could have done more learning about cybersecurity ahead of time to be like, “Those are going to come,” and to price that in. It seems very, very ecosystem dependent, rather than dependent on when we get a certain capability diffused widely, at this point. I don’t love the balance, but that’s what the evidence I’m seeing is looking at. So I do think there’s still going to be political challenges ahead for open models, even though a lot of the facts don’t support the evidence right now. 00:29:28 Nathan Lambert: I made a slide in my talk that was just screen grabs of closed models hacking companies and stuff, and there’s so many of them. I think the closed model safety notion is really, really suppressed right now, with models in training being able to hack people, people getting around the safeguards very easily and hacking companies. If you had taken us back a year ago and asked especially people at labs about that, they would have been like, “No, that’s not going to happen,” I would think. And I don’t think that is fully updated across the community very well. 00:30:07 Florian Brand: One thing that I like is that OpenAI and Anthropic seem to have updated their safeguards, because they can update them at a moment’s notice. It seems like they have rolled out an update to their safeguards that basically blocks any try to get the real exploit for PHP2Shell. If you mention, “All right, I want you to look through this code,” the safeguard will just block you off, to stop people that are just trying to get the exploit very easily in their tracks. I think that is a useful showcase for why safeguards for closed models might work better, or are working at all. 00:30:56 Nathan Lambert: I hear these things, and I hear things like researchers who are trying to do cyber research that are like, “Yeah, I haven’t been able to use any agents after Opus 4.8, because I get rolled back immediately.” You hear these, but you also do see the news that’s like, “I used Claude Code to hack OpenAI.” I think there’s still a gray area in there, where maybe the least motivated hacker is blocked by the guardrails, which is the teenager being like, “Can I steal my friend’s parents’ money out of their bank account?” That stuff’s getting stopped, blocked, which is good. But I do think open models, to a large extent, also block that type of thing. 00:31:41 Nathan Lambert: I have another rant. What do you think about Anthropic saying they’re not going to hire junior employees? 00:31:48 Florian Brand: Well, I mean, that’s great for everyone else. So they can hire all the junior employees that have very bright ideas and are very hungry to work. When we visited China, one of our takeaways was that if you just hire all the young PhD students who work their ass off all the time and are super happy to get hands on a cluster, you catch up really fast, and those people also have very bright ideas. So yeah, it’s pretty good for everyone else. 00:32:24 Nathan Lambert: That’s pretty funny. My take is, Anthropic does all this work to talk about how they want to do all this ambitious stuff to prepare society for the transition to super powerful AI, and I’m like, the number one thing you need to do is keep training people so that they are fulfilled and have things to do with their life. Yeah, taking risk on a bunch of junior employees isn’t great, but I think it’s more about the culture of too-short timelines in terms of building technology, where they’re like, “We just need to build this. I don’t have time to talk to people.” It’s another red flag on their safety culture: safety is a much broader thing. You’ve got to have your chips covering a lot of issues that matter, and a lot of these things will come back to bite them, I think, in the future. We’ve seen a lot of pushback to the labs and how they communicate things in the past, and I think this is just the next brewing issue, which is when there is a job correction or slowing jobs due to AI, and the companies, they look back six months and they’re like, “Anthropic decided not to hire any junior employees six months before a bunch of news on youth unemployment.” It’s not going to look good, and I feel like they definitely have the money. So many scientists I know that end up there like doing this. It’s a fun part of the job to have thes
We — Nathan and Florian — wanted to discuss all the messy trade-offs of a busy week in AI and ask all the pressing questions like: - Why did Kimi K3.1 Pro have a mixed reaction with such incredible benchmarks? Will it be adopted? - Were we wrong about the risks of open-weight models and cybersecurity? - Why do cyber experts discuss recent events so differently than AI experts? - What comes next in the few months? What will change our minds? We will write more about this subject, but wanted to bring you behind the curtain as we aspire to quickly learn more about the cybersecurity industry and trends across the last few years that have been leading us today. We conclude the podcast with some lighter topics, from Muse-maxing to excitement in still hiring junior technical talent. Please send us questions and topics, as we plan to record more of these podcasts going forward. Listen on Apple Podcasts, Spotify, and wherever you get your podcasts. For other Interconnects podcasts, go here. YouTube ↓ Chapters - 00:00 Kimi K3.1 Reaction - 08:15 Kimi’s open-weight cyber principles - 11:23 You can hack everything era - 15:54 PHP2Shell, LiteLLM, and the new internet - 24:00 Abliterated APIs, safeguards, and trusted access - 31:41 Anthropic not hiring juniors employees - 36:16 Florian’s Muse addiction - 45:30 Wrap-up Links we discuss - My thread with Dean Ball about the “boy who cried wolf” story of the frontier labs - Kimi K3.1 Pro announcement blog - Kimi K3.1 Pro on Artificial Analysis Intelligence Index - Kimi K3.1 on Artificial Analysis defensive cyber benchmarks - Program Bench - Open-weight Mythos would’ve been fine take - PHP “hasn’t caused a disaster yet” update - Abliteration.ai serving abliterated models API - Morph serving abliterated DeepSeek via API - Joshua Saxe’s Substack on AI and cybersecurity - Azeem Azhar, “The internet’s Wuhan moment” - Anthropic, “How employment is changing at Anthropic” - LiteLLM security incident report Transcript 00:00:00 Nathan Lambert: We are back with another Interconnects podcast. We are with Florian, eval guy, recently moved to SF. Florian, is Kimi K3.1 Pro, name is a mouthful, going to live up to the benchmark scores, for example on Artificial Analysis, or where is it going to land? 00:00:17 Florian Brand: In the past, the Kimi models always lived up to the expectations, but this release is kind of weird compared to the others. I think it is also partially because they are becoming more professional, and the previous model drops were kind of out of nowhere. This time, which also got some criticism, they launched a bigger social media campaign, which allegedly also involved paying people to post about the model without disclosing. 00:00:56 Nathan Lambert: I think a lot of companies do this and tell you not to mention it, but it’s a mess. It’s bad form. No company should do it. Essentially, one person tweeted a screenshot of the press instructions for Kimi that said, “Don’t say this is a paid promotion.” And to my knowledge, no one has corroborated it after that. 00:01:15 Florian Brand: Yeah, and there are right now a few of these model launches, including by American companies, that are just very obviously astroturfed. We have seen launches where it’s thousands of retweets and a few hundred likes, because it works so well to get the attention. Some of them disclose that it is a paid ad while others don’t. But in general, we have seen the professionalization of model launches over time, and pretty quickly. A year ago, everyone just dropped the model, then patched vLLM afterwards, so no one was actually able to run any of the open models for a while. Now you get all the inference frameworks on day one, you get all the inference providers on day one. Or, which is also what we see more and more, the model creator has a few weeks of exclusive access before they release the weights. And that’s also what’s happening with K3.1 Pro. I think it’s two weeks of exclusive access before they release the model weights. 00:02:38 Nathan Lambert: They said the weights would come out by November 4th. And the headline things... I don’t know if I’m going to be able to share screen. I don’t have this set up on this computer. But the headline is this Artificial Analysis number where it’s ahead of GPT-6 Astra. And then if you look at the Kimi K3.1 blog, they have their headline numbers, which are really strong, and then they also have some numbers later on which seem a bit more jagged. I think it was Terminal-Bench Science, which is 20 points behind GPT-6 Astra. I think there were a few others, like ProgramBench. Do you know Terminal-Bench Science and ProgramBench? Are any of these things that people think are important benchmarks for frontier models? Or how do you explain the fact that on Artificial Analysis the model is top, what, three, four? It’s the two latest Claude 5.5 models, Muse Wonder Full, and then Kimi K3.1, which looks crazy. It just looks like totally crazy scores, and I think everybody is still surprised by it. But then there’s this long tail of details that you use to evaluate a model now. I had posted a thread on Twitter of, “What are people’s impressions?” And there’s a lot of people that are like, “This seems not great,” and then some people that are like, “Yeah, it’s fine.” I don’t know, there’s a mixed bag. Relative to K3, which I think was pretty unanimously, this feels like a big model and the post-training seemed a little bit undercooked for that. Maybe this is just a tiny bit RL fried. We’ll see where we land. 00:04:26 Florian Brand: For the benchmarks, one of the things is that the benchmarks that are right now used by Artificial Analysis, they will post their new version probably very soon after we release this podcast. 00:04:43 Nathan Lambert: We can’t tell you the exact date, but both of us know. 00:04:48 Florian Brand: In general, the Artificial Analysis benchmarks are rather short in terms of time horizon, whereas something like ProgramBench is really long and also very explorative. Other benchmarks are telling the model, “All right, this is the goal, and I want you to reach it,” whereas ProgramBench is more like, “All right, here’s what I want, but you need to explore a lot.” And I think that might be the reason that we see some differences: their RL training is still more towards the exploitative side of model training versus the explorative side, where you need to find the solution somehow before you actually know what you’re doing. 00:05:41 Nathan Lambert: But if you’re using a three trillion parameter model, I feel like you kind of want it to be pretty robust at exploration. Because if you’re going to want to use a model for pretty known coding tasks, like using DeepSeek-V4.1-Flash-1017, which is not that far behind on these scores, it’s just going to be way faster. It might not be wildly different on cost per task with the actual token numbers. That’s why I think it’s pretty reasonable criticism of this Kimi release: a big model has got to be... Because it’s going to be slower, it’s got to be like Fable, which is just... I still think Fable is really nice if you’re doing very blurry things. I revert to it when checking my slides over sometimes, where I just feel like it kind of gets me a bit better, where Opus is really, really grindy. I think the open-weight models are better in this kind of grindy perspective, because you can just get them to be faster and cheaper. 00:06:44 Nathan Lambert: It seems like Kimi is gearing up for this Flash-style release with their naming change. Previous Kimi models were just K2, K2.6, K3, and now this is K3.1 Pro. So a lot of the company forecasting, investment-y side of interest on Twitter was, a lot hinges on this for Kimi, because it’s kind of true. We’re not the inference experts, but serving the model seems like you’ve got to commit way more resources to serve this three trillion versus this 700 billion parameter model. You’re going to need NVL72 racks, I’m guessing, to do it well. 00:07:21 Florian Brand: To fit the model, you need B300s, one node per model replica, which is kind of awkward, versus other models which you can even fit on H100s, or H200s rather, which are more readily available. Splitting models is always awkward compared to having one model replica per node, or using existing nodes to fit the full weights. So in that regard, three trillion is just out of, or just in the range of what you can host somewhat reasonably, but it is kind of annoying compared to all the Flash models and all the smaller models, where you can use so many of the existing GPUs that people have. 00:08:15 Nathan Lambert: Do you think Kimi’s shooting itself in the foot a little bit by saying it’s slowing its own cyber capabilities? In the blog post, I can read some of this, because it’s really good for people to say. “Open-weight principles. These evaluations frame our open-weight release decision. First, we aim not to release open weights that would extend the absolute frontier of cyber capability. K3.1 Pro’s offensive cyber capability is, by design, below what our pipeline can produce, and remains behind that of frontier proprietary models.” And they go on to say... I don’t know if the mid-training part is in this, but essentially they manipulated the mid-training data to down-weight cyber capabilities and see that that can last into the post-training. Do you think this would reflect upon the other benchmarks that they have, and maybe some of this long-horizon weakness? Or do you think that people could actually separate this? 00:09:08 Florian Brand: I think people can very well separate these things. I talked to a bunch of other people, and I think that style of release is really good. You want very broad access for everyone to find all the bugs and to patch all the bugs, which is net positive for the world. And Kimi saying they still release the capable model for the defensive side, but they don’t want to advance the offensive side. What I’ve heard from cybersecurity professionals is that exploiting and being able to go through or wreck a whole organization is way harder, because you need very long-horizon capabilities, and you need to persist through all the stages that you penetrate. That’s where closed models, especially GPT-5.6 Sol, a model that’s named often in this regard, are just way, way better than open models. And I think that is important. We want all our code to be secure, and that should happen pretty fast. But for this, you don’t need long horizon. You don’t need all the CTF-style challenges. So throwing these out of the training mix is a reasonable decision, and it helps them position against the fearmongering that once an open model with Mythos-level capabilities, which we arguably have in some regards, is released, the world will end. Which we haven’t seen so far, and I don’t think will happen in the near future. So open models get way more scrutiny than closed models in this regard, and them positioning themselves as, “We are the responsible actors,” is a smart move. 00:11:23 Nathan Lambert: Okay, we can transition into the “you can hack everything” part of this segment. There’s a lot going on here, and Florian and I are both not cyber experts, but it is worth discussing, because I’m kind of of the opinion that the cyber experts that I read talk about this from an ecosystem perspective. All measurements like vulnerability detection, time to exploitation, have been on these exponentials before ChatGPT came out, and there wasn’t a step change as AI models came out, and they focus on this ecosystem perspective. But when you come to the AI people, which is our corner of Twitter, people love to debate open versus closed models and Mythos and things like this. And then the messy middle is, what is the effect of the Daybreak programs and Mythos and things like this on broader society? I feel like I’m trying to muddle through this without getting too stuck into any one point of view. B
document fetched 2026-10-26 02:58 · canonical https://interconnects.ai/p/podcast-kimi-k31-pro-the-new-cyber