loading
loading
At first, I was planning to talk to Professor Aaronson about quantum computing since it’s his field of expertise. But in doing my research, I found that he used to be a big skeptic of AI in the 2000s. Now though, he’s completely updated his opinion based on how AI has changed his field recently. This made me curious to hear what the changes are like to him that caused him to change his mind that much. We talked about why he thinks the singularity has already begun for math, who gets credit when AI writes the proof and whether intelligence will still matter in the future. Hope you enjoy it! On another note, I created a Red Note account where I’ll start posting. It was fun to record an intro clip there in Chinese although it took me a long time. Funny thing is I tried so hard to pronounce everything perfectly that it’s a little lifeless and therefore everyone there thinks it’s an AI dub (it isn’t). Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts. Timestamps 00:34 - Why he updated so hard on AI 01:14 - The people still denying AI progress 03:53 - The singularity has already begun for math 08:30 - Will human-driven math research survive 13:12 - Who gets credit for AI-written papers 15:55 - OpenAI’s $15M Navier-Stokes proof 19:20 - What if AI proved P = NP 23:07 - Why AI is so good at finding counterexamples 26:13 - The goalposts keep moving 28:52 - Why AI disclosure matters in research 32:59 - The fake Monet experiment 35:23 - Why he blogs instead of getting rich 39:45 - Are AI labs racing too fast 42:01 - Will intelligence still matter 52:27 - Advice for students today 57:23 - Reacting live to a new AI result Transcript 00:34 — Why he updated so hard on AI Ryan: [00:34] Here’s the full episode. The reason that I wanted to talk is I thought you’d be one of the most interesting people to explain the recent events in AI, because publicly I feel like you were skeptical of it for a long time. When I was reading your blog, there’s posts going back to early 2000 talking about AI and the singularity and how far off it is. Scott: [01:00] Once I saw the trajectory, once I saw GPT-3, for example, even though it wasn’t yet useful, it was kind of a toy, I updated, I would say, as hard as I’ve ever updated on anything in my life. 01:14 — The people still denying AI progress Ryan: [01:14] Right. So, in one of your recent posts, you said one needs to be a particular kind of idiot to deny what’s going on now. Scott: [01:26] I said that because every time I talk about this, there are still all these people in very, very aggressive denial, right? I mean, even now, even now that AI is solving Millennium Prize Problems, right? And there’s going to be even more that’s released. It’s not public yet, okay? But there’s going to be hundreds of major open problems in math, computer science, physics that have been solved by AI models, that those solutions are going to get released to the world, right? [02:02] The whole nature of intellectual work is different forever, right? And still I get people saying, well, none of it really counts, right? It’s just a stochastic parrot. It’s just the next token predictor, right? It’s just outputting some slop and then humans look at the slop and find something that they call a proof or whatever thing they did. Well, then that thing wasn’t real math. [02:28] It was just a toy problem, right? Real math is still only what humans do, right? And they just keep moving the goalposts. They keep not updating, right? And I’m amazed. Part of me envies the self-confidence that they have. Ryan: [02:48] Right. Scott: [02:48] to never update no matter what happens. This summer, we saw AIs hacking out of their own training environment, right? We saw them hacking into another company’s system. We saw them cooperating with each other to do that in a totally unplanned way, right? Creating a message board. We saw them sacrificing themselves for the good of the group. And people will say, no, it’s all just hype. [03:18] It’s all just a marketing stunt. It’s all just OpenAI didn’t have good enough cybersecurity or something, right? They’ll just make up a story why it doesn’t count no matter what happens. And I expect that it’s going to be like that from now, all the way up until the singularity, right? There could be Terminators walking down the street and people will say, no, it doesn’t count, right? [03:43] It’s just, you know, they’re just token predictors that were programmed to act like Terminators. They’re not—they don’t really have an intention to kill you. 03:53 — The singularity has already begun for math Ryan: [03:53] When you say singularity, could you define that for the audience? Scott: [03:57] I mean, in math, it has precise meanings, right? It’s a point where curvature becomes infinite, for example. But in this context, it just refers to sort of a point where technological change becomes so rapid that we lose our ability to predict what happens beyond it, that afterwards we are no longer the sort of reigning intelligence on Earth. That doesn’t mean that it’s a dystopia. [04:28] It could be great, could be horrible, could be anywhere in between. But it refers to a period of, let’s say, more rapid technological change than the world has ever experienced before. And it looks like we are heading into that. To get from the solution to the Erdős unit distance problem, which is maybe the first major math problem to get solved by AI models, that was May, to the solution of Navier–Stokes, that was August. [05:01] So, okay, it took some time. It took 3 months. And now it’s taken another 2 months until pretty much all of our problems are getting solved, right? So people could say, well, it’s not literally a singularity. It takes time. But the amount of time that it’s taking is now measured in months. It’s not measured in decades, in centuries, in the kind of timescales that we’re used to for technological changes of this magnitude. Ryan: [05:30] You mentioned that you feel like the singularity has already begun in some of your writing. Scott: [05:36] For our world, it has. I mean, for the world of mathematical research, theoretical computer science research, I would say that we are already in it, in the sense that we have been dethroned as sort of the main theorem-proving entities on planet Earth, right? And it’s important for people to understand that this is not this hypothetical future thing that we’re speculating about. [06:04] This is something that happened this summer, and that we are now all trying to deal with. Right? Every meeting of mathematicians or theoretical computer scientists, whatever it was supposed to be about before, now it’s about this. It’s about how do we deal with the flood of sort of AI-enabled solutions to our longstanding open problems that are now coming at a faster rate than we can even absorb it, than we can assimilate it all, right? [06:39] How does the field continue? How does this human mathematical tradition continue? What is the future for our students? What’s the future of publication in these fields? What are we going to value going forward? Right. This is all we discuss anymore. This thing happens at different times in different fields. Right. And maybe it happened a year or two ago for software engineering. [07:14] And then someone who was doing very theoretical math research, like my friends, could say, “Oh, well, we’ll still be safe for a long time.” Right? But then it came for us. Right? It’s interestingly, it still hasn’t quite arrived for most of theoretical physics. And I think that the big difference is that in theoretical physics, it’s much less self-contained to judge what counts as a good solution to a problem, right? [07:43] We have just completely self-contained ways of checking that a solution is valid. And that makes these things very, very amenable to training on examples, to doing reinforcement learning, to getting AIs to be extremely good at these things. But when I talk to my friends in physics, they say, well, yeah, maybe we’ve got another 6 to 12 months. That’s what they’re estimating, right? And so, if you’re in one of the fields where the rising floodwaters have not yet reached, then you can tell yourself we’re going to be on dry ground forever. [08:15] Unlike those other basically trivial or machine-like fields, of course they would get flooded by this, right? Because they were never doing anything that deep, but we’re going to be fine, right? And you can keep saying that until the floodwaters come for you. 08:30 — Will human-driven math research survive Ryan: [08:30] Will human-driven mathematical research exist? Scott: [08:34] Well, that’s a very big question. I mean, partly it depends what you mean by human-driven, right? Look, if you want an optimistic story, we can look at the story of chess, right? Humans were dethroned as the top chess-playing entities on Earth 30 years ago, right? Which is like an eternity by AI standards. And some people after Deep Blue versus Kasparov predicted the death of chess, right? [09:10] People were just going to lose interest in it. And that didn’t happen at all. In fact, chess has never been more popular than it is now, right? You’ve got chess streamers, celebrities, you’ve got The Queen’s Gambit, right? People just said, we want to continue this as a human endeavor. And of course, they use AIs to train, and they use AIs to analyze the games and understand, like, what if Magnus Carlsen is playing some other grandmaster, right? [09:44] It’s like everyone else who’s watching the game actually knows vastly, vastly more about the game than they do, right? Because they can just use their solver and they can see exactly what choices did they have. What move should they have made instead of the— right? So when the game is over, instead of the rest of the world asking Magnus, like, why did you do this, right? Magnus would be asking the rest of the world, well, what should I have done? [10:10] So it’s changed the nature of chess. And yet human chess has continued to thrive. Right. And so you could say certainly there will be people who will continue to be interested in math. Think of all the people who just like learning things, even things that have long been known by other people, right? They just take courses to better themselves. [10:35] They work on math problems for fun. Okay, people will, it will continue probably as a competition. There could still be Math Olympiads and things like that. But what I doubt is that we are going to be proving theorems because we’re really needed to prove them. There’s another hard question here, which is what about setting the agenda of research? What about deciding which problems are important? [11:06] What about, and often this is the more important part of a research paper, more important than the proofs themselves, is sort of the story that someone has about why this is important, the metaphors they use, the way that they chose to frame the problem. AI can do that stuff too. You can ask GPT-6, just look at this paper, suggest some good open problems related to this paper. [11:35] It’ll do a creditable job at that, but I think that we’re still better at sort of finding interesting directions of research. Of course, there’s no objective test for that, right? The other thing is that when you look at the proofs that AI models are producing, like right now, they are terribly written. Okay. They are needlessly convoluted. They spend pages and pages on trivialities, and then they sort of skip over the essential points. [12:09] I’ve talked to students of mine who have shown me proofs that they got their AI to generate. And I said, well, I’ve got really great news for you. This write-up is garbage, right? Why is that great news? Because it means there’s still something for you to do. Understand this, clean it up, explain it for real so another person can understand it. So as long as we still have an advantage at, let’s say, setting the agenda, posing the problems, explaining things, explaining the essential points, we are still in the driver’s seat. [12:46] The question is, is that stable? Does that continue, or does it become where these models are just so much better than us, not only at proving the things, but at explaining them, at figuring out what is important, that we just say, well, unless we want to do this for our own enjoyment or maybe for sort of a sporting competition, that we should just retire and turn things over to them. 13:12 — Who gets credit for AI-written papers Ryan: [13:12] It makes me wonder, what are the new rules when you use AI to write a paper in terms of credit? What if I’m not a complexity theorist? If I draft a paper that breaks something and I send it to you, do I get credit, or how’s that work? Scott: [13:29] It’s a great question. We’ve been scrambling to figure out the rules for that since this summer. So what’s happening right now is that first of all, there’s at least twice as many papers as there were before, right? There’s just a flood of papers, sometimes very, very significant ones. Okay. They have authors, they have a list of human authors. But then at the end of every paper, there’s what’s called an AI disclosure. [13:59] Okay. By the way, I hate that these are at the end of the paper. This is often the first thing I want to know. I don’t want to have to scroll down to the bottom of it. Okay. But the AI disclosure will say something like, well, we thought of this question and then we came up with this strategy and we proved this result ourselves, but then Ashraf proved this crucial lemma and we take responsibility for the correctness of it or something like that. [14:26] But I think the evolving standard is that even if you used AI a lot, if you, first of all, understand the result, right? You can give talks about the result. Okay. You can answer questions about it. You take responsibility for it being correct, right? Then you put your name on it, and it’s your work. Okay. The harder question is what happens when those conditions are not satisfied? And that comes to what you were asking me, right? [15:02] We’re going to see more and more examples of people who just one-shot significant problems. They get a solution, which might be correct, hopefully is correct, right? And then they won’t understand anything about it, right? So it’s basically like they just put a coin into the vending machine or something like that, right? Or they just pulled the lever of the slot machine, and something came out, right? [15:29] And frankly, if one person did that, well, anyone else could have done it just as well, right? It’s like, you’re not providing great value to the world by putting this thing out if anyone else could have done the same using their own subscription to GPT. So for that kind of thing, it really seems like our whole norms around publication are going to have to shift. 15:55 — OpenAI’s $15M Navier-Stokes proof [15:55] A month or two ago was OpenAI, so ancient history now, as OpenAI published the solution to the Navier–Stokes problem, which was one of these seven Clay Millennium Problems with these million-dollar prizes. In this case, they actually did put significant — they were not experts in differential equations on this problem, but they did put significant resources. They put $15 million of compute into solving this $1 million problem. [16:28] And then they published this 166-page argument. They also had a formal certificate in this system called Lean, sort of proving that it was valid. Right. And the reaction to this was very— from the mathematical community, I think, was very negative. And this was partly because there were humans who had been sort of closing in on this problem and OpenAI was tipped off to what those humans were doing. [16:58] We still don’t know exactly how much they knew, and when they knew it, but that was what prompted them to then rush ahead with their vastly greater computing power and get there first. And people felt like, whoa, that’s not sporting. And so then, because OpenAI was burned by the negative reaction to that, since then they’ve been sitting on solutions to a ton of other. [17:27] big open problems. And now they appointed a board of experts, which includes Fields Medalists like Timothy Gowers, Edward Witten, people like that, okay, who are going to decide what to do with these, with all of these solutions. But my expectation is that at some point, very soon, in fact, all these things are going to be published. Right? Like, we’re going to start seeing hundreds of solutions to significant open problems that in most cases no human has gone through. [18:04] Okay? No human has actually checked them. I mean, in some cases there will be Lean certificates, so we sort of formally know that they’re valid. In other cases, there probably won’t even be that. I heard a joke that maybe these papers should be published and the author should be listed as GPT-6.1 or whatever. And then in the acknowledgments, it should say, “Oh, thank— I’m so grateful to this person for suggesting the problem to me.” Ryan: [18:37] Right. Scott: [18:40] But I think the more likely way that this goes is that we will just see some online repository for all these sort of AI solutions that have not yet been digested. Okay. And then you’re at human speed. Humans will go through them. They will do their best to understand the arguments, to re-explain them in better words, to simplify them, to extend them. And then they will publish papers doing that with their human names on them. [19:14] And those papers are what humans will then receive credit for. 19:20 — What if AI proved P = NP Ryan: [19:20] You mentioned OpenAI sitting on proofs of major problems. Let’s say OpenAI was sitting on a proof to the Millennium Prize problem of if P vs. NP, and suppose they proved P vs. NP. How would that change the world? Scott: [19:36] That would really change everything that we thought about computational complexity. That would sort of be a cataclysm for our universe, even greater than the AI cataclysm, you could say, that is happening right now. It’s also something that, by the way, that could massively accelerate AI. Okay. If the algorithm for solving NP problems were actually efficient in practice, which would then be a further question. [20:06] Okay. But my strong guess is that P is not equal to NP. Ryan: [20:12] Right. Scott: [20:16] I think that if and when that problem is solved, it is virtually certain at this point that AI will play a central role in the solution. Right. The question is, will humans also play a role? Right. That’s the question at this point. It’s not like, will AI be involved? It’s like, of course it will be. But if an AI proved P vs. NP, you could say, that would merely confirm what most of us have believed since the 1970s. [20:47] Okay. But I’m sure that to prove something of that magnitude, you would need a tremendous number of new ideas. And then those new ideas could be mined for a long time for all the implications that they would have for other things. Ryan: [21:04] If a proof of P vs. NP existed, how much would you say that’s worth economically? Scott: [21:12] If P were equal to NP by an algorithm that were really efficient in practice, we can estimate its economic value by just what is the economic value of everything that is happening in AI right now? Or throw in cryptocurrency also, right? Which also depends on these problems being hard to solve. So, I mean, first of all, if you had a fast-in-practice method for NP-complete problems, you could break all cryptocurrencies. [21:43] So I don’t know if that’s good or bad, but however many trillions of dollars are the total market cap of cryptocurrency, that either gets—I guess if you hide your solution and you’re very slow about it, then maybe you collect hundreds of billions of dollars. If you just reveal it all at once, then the whole market crashes. But okay, but now even bigger than that is AI, right? AI is now a multi-trillion-dollar industry, and just on this, the most rapid exponential trajectory, maybe, I guess, of any technology in human history. [22:25] Okay. Fast, in practice, a way to solve NP problems would make it massively faster to train new AI models, okay? To sort of find the parameters of this neural net that minimize the loss on this training data. You either don’t need these 100-, many 100-billion-dollar data centers anymore, or more likely in practice, you can take those data centers and use them to do something else, right? Or use them to do even more. [22:57] Let’s put it this way. The million dollars that you would get from the Clay Mathematics Institute would be an unnoticeable rounding error. 23:07 — Why AI is so good at finding counterexamples Ryan: [23:07] That question has two directions. If you think about what you would need to prove in either direction, which direction of proof would be more amenable to AI? Scott: [23:18] If P were equal to NP, then you would expect that to be very amenable to AI to find, find that because you could just need to find an algorithm, you know, and then of course give an analysis and show that it works. Okay, but the search for algorithms is very, very parallelizable, right? You could set 10,000 agents to try 10,000 different ideas for algorithms, right? And if just one of them works, then you win. [23:48] Okay, so one pattern that we’ve seen is that AI has been especially good at finding counterexamples to widely believed conjectures when those conjectures have been false. It did this. So with the Navier–Stokes, for example, where the answer was there is finite-time blowup in these equations. So there is a counterexample to the smoothness of the solutions. [24:17] It did this with what was called the Jacobian conjecture, right, which turned out to be false. And there was famously, I guess, a counterexample that was found by Fable during the World Cup. And the guy who had Fable find it just announced it via tweet. Okay. And a complete listing of the counterexample, which was really just this polynomial that takes three lines to specify, was right there in the tweet. [24:49] The Erdős unit distance problem was another example where AI found a counterexample. So I would say there’s sort of two things that AI has going for it here. One is that it’s not blinded by preconceptions the way that we are, right? It’s not discouraged by all the experts have believed this is true. I’m surely going to fail in finding a counterexample. Ryan: [25:17] No. Scott: [25:17] tell it to look for a counterexample, it’s just going to keep looking, right? And the second thing is that looking for counterexamples, as I said, is highly parallelizable. Okay? So you can often just have to try a zillion different exotic constructions. Each one might be totally unmotivated, might have some very high chance of failure, and a human would just very, very quickly want to give up in doing that. [25:44] Right. It’s just very, very unrewarding to do. And the AI doesn’t care. Right. It’ll just keep looking at counterexamples until it finds something. Ryan: [25:54] Okay. Scott: [25:54] So this is not the only thing that AI can do. It can also prove some of our longstanding, actually many of our longstanding conjectures have turned out to be true, and AI has proven them. Okay. But it’s been especially good at finding counterexamples when the counterexamples have existed. 26:13 — The goalposts keep moving Ryan: [26:13] Right. Some people use that as a way to say that the AI is not that impressive. Scott: [26:20] Yeah, right. Well, this was what I was saying before about these sort of infinite goalposts moving, right? AI, you know, struggled with arithmetic. People laughed at it, right? And they say, okay, wake me up when it can actually do math. Okay, then it does all high school math, right? All college-level problem sets. People say, well, it doesn’t count because it’s not doing real research. [26:44] Okay, then AI starts solving research problems. They say, well, it doesn’t count. Obviously it could do research problems. No one ever doubted that. Of course, those same people did doubt it 6 months ago. Wake me up when it can solve a really hard research problem like a Millennium Prize Problem, right? Then it solves a Millennium Prize Problem and they say, well, it doesn’t really count because it was just a counterexample, right? [27:06] And it’s like, no matter what happens, people will look at it after the fact and they will find a reason why it didn’t. Right. So let’s say that within the next six months or a year, as I think is probably more likely than not, suppose that another Millennium Prize Problem gets solved and suppose that the AI in this case proves that the conjectured statement was true. Right. Do we agree in advance that that will count? [27:33] Or is there going to be a different excuse then? Ryan: [28:26] Earlier, when we were talking about research, you mentioned that the AI disclosures were important, or you wish they were at the top. My question is, why does AI disclosure matter in research? 28:52 — Why AI disclosure matters in research [28:52] If you couldn’t distinguish whether I used AI and I just published some paper, why do you care if I used it? Scott: [28:59] It’s an excellent question. Because I’ll tell you why. As a theoretical computer scientist, as someone sort of who is in this game, right, I care not just about what is the answer to a question, I care about how that answer was achieved. Right? Because a person goes to a concert, they just want to hear the music. But if a professional musician goes, right, they want to know exactly what were the tricks, right? [29:31] How could I do something like that? Right. And so, it’s the same with math. You care not just about the destination, but about the journey, right? How did you get there? Okay. And part of that is just what techniques were needed within the proof. Okay. But I’m, and this is even long before AI, I was always very curious about the further question of how did these people find this proof, right? [29:59] And unfortunately, there is a long tradition in math that people will cover their tracks when they write a paper, right? They’ll say, “Here’s the proof. We now introduce the concept of a kappa comma gamma A simplicial complex that obviously is the concept that you would want for this,” with some convoluted definition. Even if you can follow all the steps mechanically, I always want to know, well, what path did you take to get to this? [30:36] Right? Because I know from experience that this kappa, gamma simplicial complex didn’t just arrive in your head. There were a huge number of false starts along the way. The authors started from what they already knew in the literature. They tried some obvious things. The obvious things didn’t work. And then that led them to something else. And I think in theoretical computer science, we do better often than in many other areas. [31:11] of actually explaining the path that we took. We have a tradition of writing long introductions where you just sort of non-rigorously explain, well, here is what you might first try, here’s why it doesn’t work, so here’s what we do instead. And then that’s kind of the appealing thing to read. And then if you want the rigorous version, then that’s all kind of tucked into long appendices. Right. And so, so I was—but I was always mostly interested in that, in that first part. [31:47] Right. In the intuition part. And in fact, often rather than even reading that part, I would rather just invite the author over to give a talk and then just sort of interrogate them during the talk and understand, like, how did they actually come up with this? Okay. And so for me, wanting to know, like, did the author use AI? How did they use AI is just part of that, with wanting to know what was the path that they took, right? [32:15] If the path involved AI, well, okay, what prompt did they give to an AI model, right? Did they just one-shot the entire problem? Or often it was more complicated than that. They had an idea, and then they fed it to AI and then AI took the next step. But it was wrong about something. And then they had to point the AI in a different direction. I want to know the story. [32:43] I want to know what happened. The analog in an experimental field would be you don’t just want to know what was the result of the experiment, right? If you’re another experimentalist, you want to know the method. You want to know how could I replicate this in my own lab? 32:59 — The fake Monet experiment Ryan: [32:59] This kind of reminds me of that famous social experiment with the Claude Monet photo where someone posted— Scott: [33:08] Oh, yes, that’s a great one. Ryan: [33:10] And then I wonder if I took an obscure human-written paper in complexity theory and I posted it, I said, “Hey, I don’t know anything about complexity theory. Here’s a paper. Please tear it to shreds on why it’s AI-generated.” Do you think that the industry would be cool? Scott: [33:27] That’s a good question. I suppose someone should try that experiment. For listeners who don’t know, right, this is, you know, I’ve used this in talks, by the way, right? But someone posted on Twitter a painting. They said, I had AI generate a painting in the style of Claude Monet. Please just explain to me exactly what makes this worse than a real Claude Monet painting. And people got hundreds of replies, right, explaining with the whole terminology of art critics. [33:58] So it’s like a derivative student work. It has no true feeling for the colors of nature, no blending, no this, no that. And only after all of that did he reveal this is an actual Claude Monet painting, right? But now an interesting question was, were these actual art critics? Like, were these people actual experts, or were these just sort of pompous people on the Internet wh
At first, I was planning to talk to Professor Aaronson about quantum computing since it’s his field of expertise. But in doing my research, I found that he used to be a big skeptic of AI in the 2000s. Now though, he’s completely updated his opinion based on how AI has changed his field recently. This made me curious to hear what the changes are like to him that caused him to change his mind that much. We talked about why he thinks the singularity has already begun for math, who gets credit when AI writes the proof and whether intelligence will still matter in the future. Hope you enjoy it! On another note, I created a Red Note account where I’ll start posting. It was fun to record an intro clip there in Chinese although it took me a long time. Funny thing is I tried so hard to pronounce everything perfectly that it’s a little lifeless and therefore everyone there thinks it’s an AI dub (it isn’t). Check out the episode wherever you get your podcasts: YouTube, Spotify, Apple Podcasts. Timestamps 00:34 - Why he updated so hard on AI 01:14 - The people still denying AI progress 03:53 - The singularity has already begun for math 08:30 - Will human-driven math research survive 13:12 - Who gets credit for AI-written papers 15:55 - OpenAI’s $15M Navier-Stokes proof 19:20 - What if AI proved P = NP 23:07 - Why AI is so good at finding counterexamples 26:13 - The goalposts keep moving 28:52 - Why AI disclosure matters in research 32:59 - The fake Monet experiment 35:23 - Why he blogs instead of getting rich 39:45 - Are AI labs racing too fast 42:01 - Will intelligence still matter 52:27 - Advice for students today 57:23 - Reacting live to a new AI result Transcript 00:34 — Why he updated so hard on AI Ryan: [00:34] Here’s the full episode. The reason that I wanted to talk is I thought you’d be one of the most interesting people to explain the recent events in AI, because publicly I feel like you were skeptical of it for a long time. When I was reading your blog, there’s posts going back to early 2000 talking about AI and the singularity and how far off it is. Scott: [01:00] Once I saw the trajectory, once I saw GPT-3, for example, even though it wasn’t yet useful, it was kind of a toy, I updated, I would say, as hard as I’ve ever updated on anything in my life. 01:14 — The people still denying AI progress Ryan: [01:14] Right. So, in one of your recent posts, you said one needs to be a particular kind of idiot to deny what’s going on now. Scott: [01:26] I said that because every time I talk about this, there are still all these people in very, very aggressive denial, right? I mean, even now, even now that AI is solving Millennium Prize Problems, right? And there’s going to be even more that’s released. It’s not public yet, okay? But there’s going to be hundreds of major open problems in math, computer science, physics that have been solved by AI models, that those solutions are going to get released to the world, right? [02:02] The whole nature of intellectual work is different forever, right? And still I get people saying, well, none of it really counts, right? It’s just a stochastic parrot. It’s just the next token predictor, right? It’s just outputting some slop and then humans look at the slop and find something that they call a proof or whatever thing they did. Well, then that thing wasn’t real math. [02:28] It was just a toy problem, right? Real math is still only what humans do, right? And they just keep moving the goalposts. They keep not updating, right? And I’m amazed. Part of me envies the self-confidence that they have. Ryan: [02:48] Right. Scott: [02:48] to never update no matter what happens. This summer, we saw AIs hacking out of their own training environment, right? We saw them hacking into another company’s system. We saw them cooperating with each other to do that in a totally unplanned way, right? Creating a message board. We saw them sacrificing themselves for the good of the group. And people will say, no, it’s all just hype. [03:18] It’s all just a marketing stunt. It’s all just OpenAI didn’t have good enough cybersecurity or something, right? They’ll just make up a story why it doesn’t count no matter what happens. And I expect that it’s going to be like that from now, all the way up until the singularity, right? There could be Terminators walking down the street and people will say, no, it doesn’t count, right? [03:43] It’s just, you know, they’re just token predictors that were programmed to act like Terminators. They’re not—they don’t really have an intention to kill you. 03:53 — The singularity has already begun for math Ryan: [03:53] When you say singularity, could you define that for the audience? Scott: [03:57] I mean, in math, it has precise meanings, right? It’s a point where curvature becomes infinite, for example. But in this context, it just refers to sort of a point where technological change becomes so rapid that we lose our ability to predict what happens beyond it, that afterwards we are no longer the sort of reigning intelligence on Earth. That doesn’t mean that it’s a dystopia. [04:28] It could be great, could be horrible, could be anywhere in between. But it refers to a period of, let’s say, more rapid technological change than the world has ever experienced before. And it looks like we are heading into that. To get from the solution to the Erdős unit distance problem, which is maybe the first major math problem to get solved by AI models, that was May, to the solution of Navier–Stokes, that was August. [05:01] So, okay, it took some time. It took 3 months. And now it’s taken another 2 months until pretty much all of our problems are getting solved, right? So people could say, well, it’s not literally a singularity. It takes time. But the amount of time that it’s taking is now measured in months. It’s not measured in decades, in centuries, in the kind of timescales that we’re used to for technological changes of this magnitude. Ryan: [05:30] You mentioned that you feel like the singularity has already begun in some of your writing. Scott: [05:36] For our world, it has. I mean, for the world of mathematical research, theoretical computer science research, I would say that we are already in it, in the sense that we have been dethroned as sort of the main theorem-proving entities on planet Earth, right? And it’s important for people to understand that this is not this hypothetical future thing that we’re speculating about. [06:04] This is something that happened this summer, and that we are now all trying to deal with. Right? Every meeting of mathematicians or theoretical computer scientists, whatever it was supposed to be about before, now it’s about this. It’s about how do we deal with the flood of sort of AI-enabled solutions to our longstanding open problems that are now coming at a faster rate than we can even absorb it, than we can assimilate it all, right? [06:39] How does the field continue? How does this human mathematical tradition continue? What is the future for our students? What’s the future of publication in these fields? What are we going to value going forward? Right. This is all we discuss anymore. This thing happens at different times in different fields. Right. And maybe it happened a year or two ago for software engineering. [07:14] And then someone who was doing very theoretical math research, like my friends, could say, “Oh, well, we’ll still be safe for a long time.” Right? But then it came for us. Right? It’s interestingly, it still hasn’t quite arrived for most of theoretical physics. And I think that the big difference is that in theoretical physics, it’s much less self-contained to judge what counts as a good solution to a problem, right? [07:43] We have just completely self-contained ways of checking that a solution is valid. And that makes these things very, very amenable to training on examples, to doing reinforcement learning, to getting AIs to be extremely good at these things. But when I talk to my friends in physics, they say, well, yeah, maybe we’ve got another 6 to 12 months. That’s what they’re estimating, right? And so, if you’re in one of the fields where the rising floodwaters have not yet reached, then you can tell yourself we’re going to be on dry ground forever. [08:15] Unlike those other basically trivial or machine-like fields, of course they would get flooded by this, right? Because they were never doing anything that deep, but we’re going to be fine, right? And you can keep saying that until the floodwaters come for you. 08:30 — Will human-driven math research survive Ryan: [08:30] Will human-driven mathematical research exist? Scott: [08:34] Well, that’s a very big question. I mean, partly it depends what you mean by human-driven, right? Look, if you want an optimistic story, we can look at the story of chess, right? Humans were dethroned as the top chess-playing entities on Earth 30 years ago, right? Which is like an eternity by AI standards. And some people after Deep Blue versus Kasparov predicted the death of chess, right? [09:10] People were just going to lose interest in it. And that didn’t happen at all. In fact, chess has never been more popular than it is now, right? You’ve got chess streamers, celebrities, you’ve got The Queen’s Gambit, right? People just said, we want to continue this as a human endeavor. And of course, they use AIs to train, and they use AIs to analyze the games and understand, like, what if Magnus Carlsen is playing some other grandmaster, right? [09:44] It’s like everyone else who’s watching the game actually knows vastly, vastly more about the game than they do, right? Because they can just use their solver and they can see exactly what choices did they have. What move should they have made instead of the— right? So when the game is over, instead of the rest of the world asking Magnus, like, why did you do this, right? Magnus would be asking the rest of the world, well, what should I have done? [10:10] So it’s changed the nature of chess. And yet human chess has continued to thrive. Right. And so you could say certainly there will be people who will continue to be interested in math. Think of all the people who just like learning things, even things that have long been known by other people, right? They just take courses to better themselves. [10:35] They work on math problems for fun. Okay, people will, it will continue probably as a competition. There could still be Math Olympiads and things like that. But what I doubt is that we are going to be proving theorems because we’re really needed to prove them. There’s another hard question here, which is what about setting the agenda of research? What about deciding which problems are important? [11:06] What about, and often this is the more important part of a research paper, more important than the proofs themselves, is sort of the story that someone has about why this is important, the metaphors they use, the way that they chose to frame the problem. AI can do that stuff too. You can ask GPT-6, just look at this paper, suggest some good open problems related to this paper. [11:35] It’ll do a creditable job at that, but I think that we’re still better at sort of finding interesting directions of research. Of course, there’s no objective test for that, right? The other thing is that when you look at the proofs that AI models are producing, like right now, they are terribly written. Okay. They are needlessly convoluted. They spend pages and pages on trivialities, and then they sort of skip over the essential points. [12:09] I’ve talked to students of mine who have shown me proofs that they got their AI to generate. And I said, well, I’ve got really great news for you. This write-up is garbage, right? Why is that great news? Because it means there’s still something for you to do. Understand this, clean it up, explain it for real so another person can understand it. So as long as we still have an advantage at, let’s s
document fetched 2026-10-24 10:06 · canonical https://developing.dev/p/e070-scott-aaronson