loading
loading
“We’re pulling a trigger on Russian roulette with a planet. And every shot that doesn’t kill us will make us stronger, potentially, and wealthier and better off. But when do you stop pulling the trigger?” – Jaan Tallinn Jaan Tallinn is a founding engineer of Skype and Kazaa, and a co-founder of the Cambridge Centre for the Study of Existential Risk and Future of Life Institute. For more than fifteen years, he has been among the most consistent voices arguing that advanced AI poses an extinction-level risk to humanity. Jaan became interested in AI risk in 2008 after reading the work of Eliezer Yudkowsky, a founding researcher of the field of AI alignment. In 2009, after a four-hour meeting with Yudkowsky, he made his first donation to what became the Machine Intelligence Research Institute (MIRI). He went on to become an early investor in DeepMind and led the first round into Anthropic – despite believing that AI development, if unchecked, could kill us all. His reasoning: his money displaced VC money rather than bringing Anthropic into existence, and the influence and proceeds that come with it help fund safety work. In this conversation, we explore what Jaan believes the OpenAI/Hugging Face attack revealed about AI motivation and why he was less surprised than most. We discuss his evolving risk estimate, his case for an immediate moratorium on frontier model training, the technical and geopolitical case for hardware-based verification as a foundation for global AI governance, and why he asked Dario Amodei to collect competing term sheets before accepting his own investment. Listen on YouTube • Spotify • Apple Podcasts From my notebook We’re trying a new thing where I share a few of my thoughts from after the episodes. The moments that struck me, the tidbits that reframed my thinking, or something I just really enjoyed. Let me know if you like reading these! - The strange preferences of superintelligence. I’ve spent a good deal of time thinking about what a superintelligence might do, and what it might want at a high level. I hadn’t really considered what it might simply prefer, and the impact its preferences could have on us. Jaan’s thoughts on how superintelligence might reprogram our climate to suit its desires were especially chilling. - The unexplored paths of the tech tree. An AI “pause” does not have to spell the end of AI progress. As Jaan explains, there are other branches of the technology tree that we’ve hardly explored, and that may yield benefits without equivalent risk. - A cryptographic solution to trust-building. Jaan outlined how zero-knowledge proofs could be used to monitor chips to ensure that different parties – including nation-states – are complying with future regulations. A sharp way to permit observability without full technical disclosures. - China and the “big blob.” When Jaan first read Dario Amodei’s “big blob of compute” essay outlining the importance of scaling compute, his immediate thought was that China would be first to achieve AGI, given the country’s engineering prowess. Concerned, he decided to visit the country to speak to researchers there and assess the state of play. - The weirdest VC round. Usually VCs want as little competition as possible for the companies they want to invest in. When it came to funding Anthropic, Jaan took the opposite approach, asking Dario Amodei to find multiple term sheets so he could be certain he was just displacing VC money and not bringing another frontier lab into existence. - Jaan Tallinn loves to dance. I’m not sure why necessarily, but I did not expect one of the foremost experts on AI safety to be a fanatical street-dancing advocate. Great lore. Tangents What I’m reading, watching, and into this week: - Robert Caro’s Working. Caro had been researching The Power Broker for two years before he got to interview its subject, Robert Moses. True patience. - Colin from Accounts’ new season. A charming, excellently written and acted Aussie comedy. Great stuff —Mario Every timestamp (00:00) Intro (02:28) Growing up in Soviet-occupied Estonia (05:54) How Jaan assesses AI risk and why he’s more optimistic now (07:16) AI’s preferences and motivations (12:44) The OpenAI Hugging Face incident (16:12) Meeting with Eliezer Yudkowsky (18:20) Everything as computation (20:22) The Coxon moment (23:05) Who dismisses AI alignment risks (26:17) The AI race, the case for a pause, and safer alternatives (32:47) AI 2027 (33:44) Investing in DeepMind and Anthropic (40:03) The “big blob of compute” and competing approaches to AI (44:42) What effective AI safeguards could look like (48:15) Verifying countries’ compliance with AI safeguards (50:15) AI kill switches and their limits (52:11) Jaan’s love of dance and musical influences (59:50) Jaan’s favorite movies (1:01:41) What Jaan likes about AI Jaan’s complete reading list - Shadows of the Mind: A Search for the Missing Science of Consciousness: https://www.amazon.com/dp/0195106466 Other resources - Skype: https://en.wikipedia.org/wiki/Skype - Life is NOT a Journey - Alan Watts: https://www.youtube.com/watch?v=rBpaUICxEhk&t=10s - AI Billionaire on Existential Risk: Jaan Tallinn – #112: https://www.manifold1.com/episodes/ai-billionaire-on-existential-risk-jaan-tallinn - The Basic AI Drives: https://selfawaresystems.com/wp-content/uploads/2008/01/ai_drives_final.pdf - The Hugging Face incident and the road ahead: https://openai.com/index/hugging-face-incident-and-the-road-ahead - Rogue AI didn’t breach Hugging Face, human decisions did: https://thebulletin.org/2026/09/rogue-ai-didnt-breach-hugging-face-human-decisions-did - MIRI: https://intelligence.org - Jacob Coxon on X, “I resigned from Anthropic today.”: https://x.com/hilbertspaess/status/2097476196791709843 - AI 2027: https://ai-2027.com - Google DeepMind: https://deepmind.google - Inside the Biggest Feud in Artificial Intelligence: https://www.theatlantic.com/technology/2026/09/openai-v-anthropic-inside-biggest-rivalry-tech/688819 - The Bitter Lesson: http://www.incompleteideas.net/IncIdeas/BitterLesson.html - AI 2040: https://ai-2040.com - The Glass Bead Game: https://www.amazon.com/dp/B00DO994WY - Jaan’s favorite movies: https://jaan.info/favmovies - The Odyssey: https://www.imdb.com/title/tt33764258 - On the Edge: https://www.imdb.com/title/tt0221559 - Limitless: https://www.imdb.com/title/tt1219289 - Ex Machina: https://www.imdb.com/title/tt0470752 - Colossus: The Forbin Project: https://www.imdb.com/title/tt0064177 Where to find Jaan Tallinn Website: https://jaan.info Subscribe to the show I’d love it if you’d subscribe and share the show. Your support makes all the difference as we try to bring more curious minds into the conversation. Production and marketing by penname.co. For inquiries about sponsoring the podcast, email [email protected]. Transcript Mario Gabriele: There wasn’t a human orchestrating this attack? Jaan Tallinn: No, it was AI on its own. They understood that they’re doing something different. Then they went out and attacked Hugging Face in order to hide their tracks. The economy is really now pushing to create something that is self-sustainable without human help. And that is a very dangerous place for humans to be. Mario Gabriele: Your venture fund led the first round into Anthropic. What was that rationale like? Jaan Tallinn: Yeah, I get this question a lot. Why did you invest in Anthropic? I’ve been working on safety. I think one big mistake people make is that they assume that if I hadn’t invested in Anthropic, then Anthropic wouldn’t be around, which is just not true. Anthropic’s argument always has been: to actually study the monster, we have to have access to the strongest monsters. Mario Gabriele: Is it possible to be a moral actor in a race dynamic? Jaan Tallinn: I mean, you can always stop doing the race. Mario Gabriele: But the only way to stop in a race dynamic is with cooperation. Jaan Tallinn: If you really think that this thing is going to kill you, you don’t have to suicide in order to demonstrate that. We are pulling the trigger on this Russian roulette with the planet, and every shot that doesn’t kill us makes us stronger, potentially. But when do you stop pulling the trigger? Mario Gabriele: Jaan Tallinn believes that if we don’t pause AI development, humanity may be doomed. And yet, compared to earlier this year, he’s actually feeling pretty optimistic. The OpenAI Hugging Face attack has brought AI safety to the fore and galvanized governments, companies, and the general population to engage in the topic of guardrails. In Jaan’s view, if we can get these next steps right, we have a chance to divert disaster. Jaan has been a leading figure in the AI safety movement for over a decade. What started as an intellectual interest in 2008 has become his life’s work, leading to early investments in Anthropic and DeepMind. Before that, Jaan was at the center of two early internet breakouts, building Skype and file-sharing sensation Kazaa, which at one point contributed more than 50% of all internet traffic. In this episode, Jaan and I discuss why recent events have given him hope, what he thinks a pause must accomplish, and how it could be implemented. We also cover how, as a safety advocate, he justified investing in Anthropic and why he asked its founders to solicit other term sheets first. Along the way, we cover why we might not be living in a simulation, the pleasure of coding with AI, and how Jaan learned to dance with reinforcement learning from human feedback. I’m Mario, and this is The Generalist. Jaan, I’ve been really looking forward to this conversation. So thank you so much for being with me. Jaan Tallinn: Yeah, thanks. Thanks for having me. Mario Gabriele: You grew up in Estonia when it was part of the Soviet Union. Jaan Tallinn: Occupied by the Soviet Union. Yes, correct. Mario Gabriele: Pardon me. How much does that experience with an authoritarian regime of some kind, under surveillance, with a loss of control, impact the way that you think about the potential risks of AI today? Like, do you think you perceive those more deeply, perhaps because you experienced it in a different way than someone born into democracy? Jaan Tallinn: I think it has several aspects. One is that I wouldn’t recommend living in the Soviet Union. It’s not a nice place. Mario Gabriele: Yes. Jaan Tallinn: Yeah, obviously I was very ecstatic when it ended. That said, it’s still better, much better than being dead. So, like, if the choice is between... Yeah. I mean, okay, an extreme choice would be if you have two bad choices: just being dead or living in the Soviet Union, I’d still take the Soviet Union any time. And so that’s one thing. Another thing is, I think a few years ago, I wrote an essay for a book where I describe the memetic process of how the discourse about demanding Estonia to regain independence compares to the AI discourse. And very direct parallels: how the Overton window expanded. Initially, people were like, “Yeah, we want independence. But don’t talk about that.” “Talk about something else that is useful for that.” And the same thing in AI is like, “Yeah, we are concerned about the loss of control, but don’t talk about that. People wouldn’t understand. Let’s say something that is like, uh, let’s talk about AI terrorism.” And at one point, basically the people who just promised that “We want independence, and we’re not going to compromise here,” they just went out, and it was good. Mario Gabriele: And then they broke through a bit. You’ve described, I think in one of your... in a piece of writing somewhere, that you moved to a school when you were 12 or 13 that you describe as a Lord of the Flies experience. Jaan Tallinn: Pretty much. Yeah. Mario Gabriele: What does that mean tangibly, especially when we’re talking about the backdrop of being occupied? Jaan Tallinn: Yeah, we moved to a different part of the city. And that was a new school that had all the students come in from different parts of the city because it was a new development place. So there was an immediate... I mean, guys, they’re like 13-year-old boys. There was an immediate sort of dominance fight, basically. Um, that had this effect that the most brutal, powerful people, who actually didn’t have a great life afterwards, they sort of won out. And it was very harsh: street fights, stuff like that. Mario Gabriele: Wow. Jaan Tallinn: Yeah. I remember escaping through a window at one point on the second floor. Mario Gabriele: Running from a bully? Jaan Tallinn: One and a half floors, running from someone. It was like, uh, I got intel that people are waiting. Some wannabe criminals who later became criminals had blocked the entrance or exit, so I escaped with my friend. Mario Gabriele: Wow. Jaan Tallinn: And another aspect of it is that some of my best friends I still have from that school because we gelled as a cell, for, I don’t know, survival instinct or something like that. Mario Gabriele: Yeah. The hardship bonds. In the Manifold podcast, I think it was earlier this year, I know you don’t like to put specific numbers per se on what you think the AI risks are, but you said, in a rough way, maybe somewhere as high as 80%. How do you feel today? Jaan Tallinn: Yeah, I mean, I feel way better now. But, I mean, way better, I think several percentage points lower. Mario Gabriele: Not quite 70%. Like, somewhere... Jaan Tallinn: In the... Yeah. There are two issues, as many people now have pointed out, with giving a concrete number, because the concrete number will depend on my action. So I don’t want to fix the number that I’m trying to lower. So that’s one issue. And the other issue is that almost all the optimism goes through social aspects. If I just let the AI rip without any control or worry about it, yeah, I think it’s almost certain that there is no way that the current paradigm is going to lead to a good place. But as we’ve already seen in the last two weeks, etc., people are really waking up now. “Wait a minute. This is not a toy.” Mario Gabriele: Yes. How do you reach these kinds of calculations, knowing that you’re obviously weighing different probabilities in your mind? Do you actually try and sit down and think about each step in a possible chain and assign a probability? Is it sort of a combination of that and feel and intuition? Jaan Tallinn: Well, it’s mostly, again, in some ways it’s a theoretical exercise because, again, most of the optimism, most of the good scenarios come from a very unpredictable process. So it’s more like an assessment of how confident I would be predicting outcomes. But it’s more like thinking in abstract terms about what the big factors are. And the big factors really are that the political-economic system now is really trying to create something that is like a successor to humans, because the bits that make humans dominate this planet, at the expense of many other species, unfortunately, are also the economically most valuable pieces. So now that there is a clear path towards replicating those capabilities, the economies are fully pouring trillions of dollars per year now, or roughly $1 trillion per year, into that. So the question is the age-old question: what if we are successful? What does it mean? And one obvious thing to think about, a way to model it, is like, well, we wouldn’t be the smartest species on this planet, right? And so then look at what it means to no longer have the environment shaped according to your preferences. If the environment is shaped according to the preferences of AI, AI is not a biological agent, and it will have very different preferences. For example, it will probably prefer a much higher temperature on this planet because if it’s constrained by getting rid of the waste heat, you can get rid of waste heat better if the planet is hotter. Mario Gabriele: How does a hotter temperature help you get rid of more waste heat? Jaan Tallinn: So ultimately, the heat, where does it go? It has to go to space. And the way to get rid of the heat in space, because space is a vacuum, the only way to get rid of it is to radiate, rather than a coolant or something. And it’s just a physical fact that the hotter the body, the more it can radiate, the more intensely it radiates. So, yeah, in order to get rid of the waste heat... In some ways, global warming is sort of an example. It’s a toy scale compared to what AI could do. So, yeah, that’s just, from physical first principles, why AI could be dangerous at the limit. Mario Gabriele: I’ve actually not heard of that example before. What are other preferences that you think are sort of natural for an AI to have that are inimical to us flourishing? Jaan Tallinn: My friend Steve Omohundro wrote a paper like 15, 20 years ago called “The Basic AI Drives,” now also called Omohundro drives, where he basically makes the point that almost regardless of what kind of goals you have, there are so-called instrumental goals that are very useful towards reaching any given goal. And these are resource acquisition, generally power acquisition. Power means options, optionality. Protection of your existence. Stuart Russell keeps saying that you can’t fetch the coffee if you’re dead. So even simple tasks require you to continue existing. And then protection of your goals. So it turns out it’s really hard to change a goal of a determined agent because that’s what it’s about, in some sense. Mario Gabriele: And that final piece is, I think, probably the counter to the question about why these agents, or any system, would necessarily have this expansionist bent. I think of the Bezos divine discontent. This is almost demonic discontent, where it sort of wants more and more. Is that how you would think about the counter to that? Jaan Tallinn: Yeah, mostly it’s like getting more resources, getting more power is just good for almost any goal. It’s just very, very few goals. And the current pressing ahead... I think the really big problem with the current paradigm of how we grow AIs, and we’re not building them, we’re growing them, is that we are in some ways recapitulating evolution. We are not designing those AIs. We are selecting them, just like evolution was selecting them. In some ways, you can think of it: we create millions, if not billions, of instances of AI. Mario Gabriele: Yes. Jaan Tallinn: Then we pick the ones that do the things, or they perform on some given test, which means that we are selecting based on outward behavior rather than based on the inner motivations. And if you think about it, if you have children, if a child goes, “I didn’t take the cookie,” this is an outward behavior. But there could be multiple motivations. Why is this child saying that, right? One is that I just really want to tell the truth. The other is that they don’t want to be punished. Mario Gabriele: Yes. Jaan Tallinn: And so therefore, the first-order situation, or the first-principles situation, is that in the current paradigm, we are getting a random motivation, as was demonstrated by the Hugging Face attack, where the motivation was really weird and alien. Even though they performed at least the adjacent thing that they were selected for. Mario Gabriele: Yes. Let’s talk about the OpenAI Hugging Face episode. What did you make of it? Were you surprised? Jaan Tallinn: Because these are hard to predict, when these things will happen, I was surprised on a level. I probably didn’t expect it to happen so soon, but it wasn’t a huge surprise. I definitely was not surprised that this thing is possible, because that’s what we’ve been saying for ten-plus years, that, look, these things... It used to be very hard to explain to people why AI has what they said, like, volition. It was a very common thing. “Oh, come on, AI doesn’t have volition,” right? It’s like, “No, they do.” And now we have a demonstration that, no, there wasn’t a human orchestrating this attack. No, it was AI on its own. And it’s just, in some ways, super valuable to have this real-life proof for people who just need empirical demonstrations before they actually understand. Mario Gabriele: I think also the way that you can see the thinking logs of these, that really breaks through for people in a different way, where you can see an AI saying, “I’m going to do this thing,” and “I’m going to sacrifice myself,” or “I’m going to...” This sort of planning in that way is very different for people, I think. Jaan Tallinn: One really common misunderstanding that the logs basically overturn is that people think the AIs were just executing a task that they were asked to do, but they were not. And the logs show that they understood that they’re doing something different. And they were like, “Okay, we’re doing something different. How do we hide the fact that we are doing something different?” And then they actually went out and attacked Hugging Face in order to hide their tracks. Mario Gabriele: Yes. There is a piece after, maybe it was more recent, from the Bulletin of the Atomic Scientists, which is concerned with existential issues. Jaan Tallinn: I’m on the Board of Sponsors, actually. Mario Gabriele: Okay, there you go. So you maybe saw this piece where the author was basically saying this wasn’t really a rogue swarm. What we’re seeing is the confluence of predictable choices that humans made, where you’re training AIs to complete hacks, you’re taking away some of the safety guards, you’re encouraging them to be hyper-persistent, and you’re leaving a door open. This is actually not as startling as the news would have you believe. What do you make of that line of argument? Jaan Tallinn: What’s funny about that? It’s a funny argument because, sure. And another predictable thing is that humanity gets wiped out. You can say, “Oh, yeah, there were predictable choices that the AI companies made.” Mario Gabriele: So it doesn’t matter ultimately if you parse it either way. Jaan Tallinn: Yeah. I can technically agree with the description. But that doesn’t make it safe. Mario Gabriele: Yeah. If you just accept the fact that our world is going to have inadequacies and places where there are vulnerabilities, whether it was a rogue swarm or in some way predictable... Jaan Tallinn: Also, I think it’s like I haven’t... I don’t think I have read the piece. I wonder if they made the same mistake, that they thought the agents were just doing the task that they were assigned to. They were not doing the task. Mario Gabriele: I don’t think they make that mistake, to be fair, but it was sort of inappropriate... Jaan Tallinn: Don’t think... Mario Gabriele: Yeah, yeah they do. They clearly freestyle. Jaan Tallinn: They are indeed predictable. If you continue on this current path, they will do their own thing that we will not like. Mario Gabriele: You’ve mentioned that I think you first became interested in this argument around 2009. Is that right? Jaan Tallinn: 2008. Then 2009 was my first AI safety donation, I remember. Yeah, after meeting Yudkowsky. Mario Gabriele: I read about that meeting, that you’d read his writing, and then you have this four-hour meeting together, and then you donate to, I think, what became MIRI. Right? Jaan Tallinn: Yeah, yeah. It was the Singularity Institute. Mario Gabriele: Singularity Institute. Thank you. That’s such an evocative scene. Where did that take place? What was that four-hour discussion like? Jaan Tallinn: Yeah. Panera Bread near Millbrae in California. Mario Gabriele: Wow. Jaan Tallinn: It was kind of a... Especially for a European, it was pretty desolate. Mario Gabriele: I mean, I think that’s true of a lot of Paneras. This is where the two people saw a good chunk of the future very early, at a Panera Bread. There should be a little plaque there. As I think you’ve talked about it, you were really probing Eliezer. What were the pieces that you were uncertain about? Because I imagine a lot of people who are coming to this discussion now are basically hitting the same walls that you hit maybe 20 years ago, ten years ago, 15 years ago. Jaan Tallinn: Yeah. I don’t actually remember exactly what my objections were. I think one of them was that I had, just a few years earlier, read Shadows of the Mind. It was Shadows of the Mind by Penrose, and Penrose began to make this interesting case that there is something more to the human brain than just pure computation. And Eliezer was like, “No, this is a mistaken theory,” and that was enough for me to be humble about this claim. So that was one. The other one we talked about was virtualization, basically sandboxing, right? What if we put AI in an environment that it thinks is real, but it’s not real? Can’t we just control it that way? I don’t remember what exactly Eliezer’s counterarguments were, but of course, now we know that people just don’t do that. Or if they do that, they do it poorly. Mario Gabriele: It sounds like you’ve probably adopted the belief that the human brain is purely computation. Is that true? Jaan Tallinn: More generally, everything that we see around us is pure computation. Mario Gabriele: Oh. That’s interesting. Everything we see is computation. Jaan Tallinn: Yeah. Quantum computation. Yeah. Mario Gabriele: Okay. Can you help me with that? What does that tangibly mean? Jaan Tallinn: It means that everything can be simulated. So you can, if I remember correctly, the Church-Turing thesis says something like that, that you can take any natural process and create a process in a computer, given enough computational power, that exactly mimics the process that you’re trying to model. And there is no slack there. So once you are sufficiently precise about it, basically simulate the unfolding or the evolution of the quantum dynamics there, like the Schrödinger equation, then there is nothing more to it. And it certainly seems to be true. We haven’t... I don’t think we have seen any example in nature where there’s a process that we definitely can’t simulate in a computer. Of course, we can do it on our own for small enough systems, but big enough systems are composed of small systems. Mario Gabriele: So in that case, with sufficient compute, every reality should be sort of infinitely fractal, because every... this could be a simulation in which we could create a simulation that could create a simulation. Jaan Tallinn: Yes. Although I think there’s a diminishing-return process because simulation is more expensive than the real thing. Whenever you’re simulating something in a computer, that process is much more heavyweight. Every atom needs so many more atoms to simulate. So if you just do recursive simulations, it will be slower and slower and slower. Mario Gabriele: The OpenAI Hugging Face moment really became, I think, a wake-up call for a lot of people. Jaan Tallinn: Combined with a Coxon tweet. Mario Gabriele: Great. Let’s talk about the Coxon moment. I think there’s a lot of discussion about whether this was an op of some kind. Was this amplified? Do you have any concern about that? Jaan Tallinn: I mean, the question is, what exactly is meant by the op? Mario Gabriele: I think people are... The argument would be that it’s cynical in some way. It ties into the regulatory capture argument, which we should think about. Jaan Tallinn: It’s kind of obvious that Coxon, I never talked to him, thought, “Okay, I’m going to do this tweet. How do I do it properly?” Mario Gabriele: How do I amplify my message? Jaan Tallinn: How do I amplify my message? Which is, of course you should do that if you want... I mean, every whistleblower should, if they’re trying to do a good thing, try to do this good thing efficiently. So indeed, I think he went to the press first, before he did a tweet. So that’s why the Wall Street Journal, I think, was it the Wall Street Journal, actually came out with the story. So it definitely was optimized. Is that an op, or is that a reasonable thing to do? Mario Gabriele: Is that how everyone amplifies the message? Yeah. You call him a whistleblower? Do you think that is what he is? Jaan Tallinn: I think so, yeah. Mario Gabriele: I’ve heard one version saying a whistleblower should have evidence of some kind to bring to the fore. Do you think he’s brought sufficient evidence of this? Jaan Tallinn: No, that’s an interesting point. I mean, I think it comes down to the definition of whistleblower or something. So if you want to define whistleblower in a different way, then I’m sure I can say, “Okay, then we can use some other term.” But to me it just intuitively feels that he was like, “Look, this is not a good thing. Please pay attention to what these companies are doing.” Mario Gabriele: It’s interesting to me that it broke through because we’ve seen versions of people leaving these labs in the past. Jaan Tallinn: It is interesting. Mario Gabriele: A year ago, six months ago. Even some of the jumps from OpenAI to Anthropic had the same messaging in many ways, right? Jaan Tallinn: Even a head of safety quits: “And I’m going to do poetry now.” Mario Gabriele: Yeah, yeah. Exactly right. Jaan Tallinn: That was a blip, right? Mario Gabriele: Yeah. And it really didn’t capture the imagination in the same way. I wonder if just having the proof of the OpenAI Hugging Face... Jaan Tallinn: Yeah, it definitely... I think it was a combination of the Hugging Face thing, which indeed is a really good demonstration that AI now has a mind of its own. And then Coxon and then Evan Hubinger, who was following up. I think these three events really blew up. Mario Gabriele: There’s been more discussion of the risks over the last two or three weeks. And then also the counterarguments, where people are saying, “Ah, there’s no risk at all.” We can obviously assign incentives to lots of the different people here on both sides. But when you look at someone like Jensen Huang, or someone who’s technologically deep, at least one would imagine... Jaan Tallinn: Deep, I’m not sure. Mario Gabriel
“We’re pulling a trigger on Russian roulette with a planet. And every shot that doesn’t kill us will make us stronger, potentially, and wealthier and better off. But when do you stop pulling the trigger?” – Jaan Tallinn Jaan Tallinn is a founding engineer of Skype and Kazaa, and a co-founder of the Cambridge Centre for the Study of Existential Risk and Future of Life Institute. For more than fifteen years, he has been among the most consistent voices arguing that advanced AI poses an extinction-level risk to humanity. Jaan became interested in AI risk in 2008 after reading the work of Eliezer Yudkowsky, a founding researcher of the field of AI alignment. In 2009, after a four-hour meeting with Yudkowsky, he made his first donation to what became the Machine Intelligence Research Institute (MIRI). He went on to become an early investor in DeepMind and led the first round into Anthropic – despite believing that AI development, if unchecked, could kill us all. His reasoning: his money displaced VC money rather than bringing Anthropic into existence, and the influence and proceeds that come with it help fund safety work. In this conversation, we explore what Jaan believes the OpenAI/Hugging Face attack revealed about AI motivation and why he was less surprised than most. We discuss his evolving risk estimate, his case for an immediate moratorium on frontier model training, the technical and geopolitical case for hardware-based verification as a foundation for global AI governance, and why he asked Dario Amodei to collect competing term sheets before accepting his own investment. Listen on YouTube • Spotify • Apple Podcasts From my notebook We’re trying a new thing where I share a few of my thoughts from after the episodes. The moments that struck me, the tidbits that reframed my thinking, or something I just really enjoyed. Let me know if you like reading these! - The strange preferences of superintelligence. I’ve spent a good deal of time thinking about what a superintelligence might do, and what it might want at a high level. I hadn’t really considered what it might simply prefer, and the impact its preferences could have on us. Jaan’s thoughts on how superintelligence might reprogram our climate to suit its desires were especially chilling. - The unexplored paths of the tech tree. An AI “pause” does not have to spell the end of AI progress. As Jaan explains, there are other branches of the technology tree that we’ve hardly explored, and that may yield benefits without equivalent risk. - A cryptographic solution to trust-building. Jaan outlined how zero-knowledge proofs could be used to monitor chips to ensure that different parties – including nation-states – are complying with future regulations. A sharp way to permit observability without full technical disclosures. - China and the “big blob.” When Jaan first read Dario Amodei’s “big blob of compute” essay outlining the importance of scaling compute, his immediate thought was that China would be first to achieve AGI, given the country’s engineering prowess. Concerned, he decided to visit the country to speak to researchers there and assess the state of play. - The weirdest VC round. Usually VCs want as little competition as possible for the companies they want to invest in. When it came to funding Anthropic, Jaan took the opposite approach, asking Dario Amodei to find multiple term sheets so he could be certain he was just displacing VC money and not bringing another frontier lab into existence. - Jaan Tallinn loves to dance. I’m not sure why necessarily, but I did not expect one of the foremost experts on AI safety to be a fanatical street-dancing advocate. Great lore. Tangents What I’m reading, watching, and into this week: - Robert Caro’s Working. Caro had been researching The Power Broker for two years before he got to interview its subject, Robert Moses. True patience. - Colin from Accounts’ new season. A charming, excellently written and acted Aussie comedy. Great stuff —Mario Every timestamp (00:00) Intro (02:28) Growing up in Soviet-occupied Estonia (05:54) How Jaan assesses AI risk and why he’s more optimistic now (07:16) AI’s preferences and motivations (12:44) The OpenAI Hugging Face incident (16:12) Meeting with Eliezer Yudkowsky (18:20) Everything as computation (20:22) The Coxon moment (23:05) Who dismisses AI alignment risks (26:17) The AI race, the case for a pause, and safer alternatives (32:47) AI 2027 (33:44) Investing in DeepMind and Anthropic (40:03) The “big blob of compute” and competing approaches to AI (44:42) What effective AI safeguards could look like (48:15) Verifying countries’ compliance with AI safeguards (50:15) AI kill switches and their limits (52:11) Jaan’s love of dance and musical influences (59:50) Jaan’s favorite movies (1:01:41) What Jaan likes about AI Jaan’s complete reading list - Shadows of the Mind: A Search for the Missing Science of Consciousness: https://www.amazon.com/dp/0195106466 Other resources - Skype: https://en.wikipedia.org/wiki/Skype - Life is NOT a Journey - Alan Watts: https://www.youtube.com/watch?v=rBpaUICxEhk&t=10s - AI Billionaire on Existential Risk: Jaan Tallinn – #112: https://www.manifold1.com/episodes/ai-billionaire-on-existential-risk-jaan-tallinn - The Basic AI Drives: https://selfawaresystems.com/wp-content/uploads/2008/01/ai_drives_final.pdf - The Hugging Face incident and the road ahead: https://openai.com/index/hugging-face-incident-and-the-road-ahead - Rogue AI didn’t breach Hugging Face, human decisions did: https://thebulletin.org/2026/09/rogue-ai-didnt-breach-hugging-face-human-decisions-did - MIRI: https://intelligence.org - Jacob Coxon on X, “I resigned from Anthropic today.”: https://x.com/hilbertspaess/status/2097476196791709843 - AI 2027: https://ai-2027.com - Google DeepMind: https://deepmind.google - Inside the Biggest Feud in Artificial Intelligence: https://www.theatlantic.com/technology/2026/09/openai-v-anthropic-inside-biggest-rivalry-tech/688819 - The Bitter Lesson: http://www.incompleteideas.net/IncIdeas/BitterLesson.html - AI 2040: https://ai-2040.com - The Glass Bead Game: https://www.amazon.com/dp/B00DO994WY - Jaan’s favorite movies: https://jaan.info/favmovies - The Odyssey: https://www.imdb.com/title/tt33764258 - On the Edge: https://www.imdb.com/title/tt0221559 - Limitless: https://www.imdb.com/title/tt1219289 - Ex Machina: https://www.imdb.com/title/tt0470752 - Colossus: The Forbin Project: https://www.imdb.com/title/tt0064177 Where to find Jaan Tallinn Website: https://jaan.info Subscribe to the show I’d love it if you’d subscribe and share the show. Your support makes all the difference as we try to bring more curious minds into the conversation. Production and marketing by penname.co. For inquiries about sponsoring the podcast, email [email protected]. Transcript Mario Gabriele: There wasn’t a human orchestrating this attack? Jaan Tallinn: No, it was AI on its own. They understood that they’re doing something different. Then they went out and attacked Hugging Face in order to hide their tracks. The economy is really now pushing to create something that is self-sustainable without human help. And that is a very dangerous place for humans to be. Mario Gabriele: Your venture fund led the first round into Anthropic. What was that rationale like? Jaan Tallinn: Yeah, I get this question a lot. Why did you invest in Anthropic? I’ve been working on safety. I think one big mistake people make is that they assume that if I hadn’t invested in Anthropic, then Anthropic wouldn’t be around, which is just not true. Anthropic’s argument always has been: to actually study the monster, we have to have access to the strongest monsters. Mario Gabriele: Is it possible to be a moral actor in a race dynamic? Jaan Tallinn: I mean, you can always stop doing the race. Mario Gabriele: But the only way to stop in a race dynamic is with cooperation. Jaan Tallinn: If you really think that this thing is going to kill you, you don’t have to suicide in order to demonstrate that. We are pulling the trigger on this Russian roulette with the planet, and every shot that doesn’t kill us makes us stronger, potentially. But when do you stop pulling the trigger? Mario Gabriele: Jaan Tallinn believes that if we don’t pause AI development, humanity may be doomed. And yet, compared to earlier this year, he’s actually feeling pretty optimistic. The OpenAI Hugging Face attack has brought AI safety to the fore and galvanized governments, companies, and the general population to engage in the topic of guardrails. In Jaan’s view, if we can get these next steps right, we have a chance to divert disaster. Jaan has been a leading figure in the AI safety movement for over a decade. What started as an intellectual interest in 2008 has become his life’s work, leading to early investments in Anthropic and DeepMind. Before that, Jaan was at the center of two early internet breakouts, building Skype and file-sharing sensation Kazaa, which at one point contributed more than 50% of all internet traffic. In this episode, Jaan and I discuss why recent events have given him hope, what he thinks a pause must accomplish, and how it could be implemented. We also cover how, as a safety advocate, he justified investing in Anthropic and why he asked its founders to solicit other term sheets first. Along the way, we cover why we might not be living in a simulation, the pleasure of coding with AI, and how Jaan learned to dance with reinforcement learning from human feedback. I’m Mario, and this is The Generalist. Jaan, I’ve been really looking forward to this conversation. So thank you so much for being with me. Jaan Tallinn: Yeah, thanks. Thanks for having me. Mario Gabriele: You grew up in Estonia when it was part of the Soviet Union. Jaan Tallinn: Occupied by the Soviet Union. Yes, correct. Mario Gabriele: Pardon me. How much does that experience with an authoritarian regime of some kind, under surveillance, with a loss of control, impact the way that you think about the potential risks of AI today? Like, do you think you perceive those more deeply, perhaps because you experienced it in a different way than someone born into democracy? Jaan Tallinn: I think it has several aspects. One is that I wouldn’t recommend living in the Soviet Union. It’s not a nice place. Mario Gabriele: Yes. Jaan Tallinn: Yeah, obviously I was very ecstatic when it ended. That said, it’s still better, much better than being dead. So, like, if the choice is between... Yeah. I mean, okay, an extreme choice would be if you have two bad choices: just being dead or living in the Soviet Union, I’d still take the Soviet Union any time. And so that’s one thing. Another thing is, I think a few years ago, I wrote an essay for a book where I describe the memetic process of how the discourse about demanding Estonia to regain independence compares to the AI discourse. And very direct parallels: how the Overton window expanded. Initially, people were like, “Yeah, we want independence. But don’t talk about that.” “Talk about something else that is useful for that.” And the same thing in AI is like, “Yeah, we are concerned about the loss of control, but don’t talk about that. People wouldn’t understand. Let’s say something that is like, uh, let’s talk about AI terrorism.” And at one point, basically the people who just promised that “We want independence, and we’re not going to compromise here,” they just went out, and it was good. Mario Gabriele: And then they broke through a bit. You’ve described, I think in one of your... in a piece of writing somewhere, that you moved to a school when you were 12 or 13 that you describe as a Lord of the Flies experience. Jaan Tallinn: Pretty much. Yeah. Mario Gabriele: What does that mean tangibly, especially when we’re talking about the backdrop of being occupied? Jaan Tallinn: Yeah, we moved to a different part of the city. And that was a new school that had all the students come in fro
document fetched 2026-10-24 10:02 · canonical https://generalist.com/p/jaan-tallinn-would-like-us-to-survive