loading
loading
Ramez Naam is, in my opinion, one of the best futurists alive. But although he has a Substack (which you should definitely follow), he doesn’t write nearly enough. So I’ve been encouraging him to write more by cross-posting his posts on my blog. Somewhat to my surprise, Ramez has turned out to be a lot more of an AI skeptic than I would have though. He’s certainly not dismissive of the technology, but he disagrees with many of the more expansive and dramatic claims coming out of the big labs (especially Anthropic). In his first post in this series, he expressed skepticism that recursive self-improvement (RSI) would lead to a fast takeoff of AI capabilities in the near future: In this post, he expresses skepticism about the potential for agent swarms — large numbers of AI agents working in parallel — to offer dramatic increases in AI capabilities. Agent swarms have been in the news a lot, both because of the Hugging Face attack, and OpenAI’s solution of the famous Navier-Stokes problem is also widely attributed to a large swarm. But Ramez argues that parallelization is just another kind of input that suffers from diminishing returns, and that we should therefore temper our expectations about what agents swarms are able to achieve. As was the case with his previous post, I’m not sure how much this matters, when the capabilities of the underlying models — swarm or no swarm — are still improving at such a steady and impressive rate. But it’s a very intellectually interesting argument, and Ramez can be very persuasive. In my previous post, we asked whether AIs could improve themselves all the way to absurdly high levels of superintelligence. My conclusion: today’s feedback loop is probably too weak, because improving AI faces diminishing returns at every step. Even fully autonomous AIs doing all the research and coding don’t make those diminishing returns disappear. We should keep updating that conclusion as real-world parameters and data change. But we didn’t look closely at another possibility: running millions or billions of AI minds in parallel. How much do they help? That’s what we’ll look at here. TL;DR Plenty of people have modeled how more AI agents working in parallel could fuel recursive self-improvement. But the data we have today tell a different story. More agents are useful for speed: they can solve many problems faster. They’re less useful for boosting AI intelligence. Multi-agent gains in solving harder problems often diminish incredibly quickly and may even saturate. Whether multi-agent systems can produce the conceptual breakthroughs RSI might require to become self-sustaining is the harder question. Without breakthroughs in how well agents parallelize problems - something that may depend on the structure of machine learning problems themselves - it’s hard to see even millions or billions of copies of today’s models making repeated breakthroughs on the hardest problems. These are among the strongest diminishing returns we’ve looked at. Of course, these diminishing returns may change. And we have less experimental data to go off of today than we’d like. We’ll need to recalibrate as new evidence comes in. I’ll end with the things we should track and my predictions about a natural experiment from newly released Anthropic data. Below is the two-image summary: The experimental observations of how large numbers of agents scale to solve harder problems (what I’m referring to as intelligence) are far far lower than the assumptions in studies that predict that AI recursive self-improvement will lead to an intelligence explosion. The first image is at the scale of 100 parallel AI agents. The second is at the scale of a billion AI parallel agents. Real world experiments show dramatically lower intelligence gains from parallel agents than hypothetical AI models. Extrapolated out to a billion agents, the intelligence gain in hypothetical models is orders of magnitude beyond real world experiment gain curves. What we’ve already covered regarding RSI and diminishing returns To recap the last post: recursive self-improvement (RSI) means AI improvements feeding back into the next round of AI research. Four findings stood out: - Real AI research is harder than benchmark scores suggest. OpenAI’s models needed human help on much shorter research tasks than the coding benchmarks led us to expect. - More AI activity produces much smaller gains in useful work. OpenAI used 124 times as many tokens per researcher but ran only 1.6 times as many experiments per researcher. - Maintaining progress requires ever more resources. Training and thinking time face steep diminishing returns. Anthropic reports that more powerful AI has been key to maintaining the current pace of AI progress. - The measured software feedback loop doesn’t yet look self-sustaining. Our tentative calibration put it roughly five to ten times below the strength needed. RSI doesn’t necessarily mean runaway superintelligence I want to reiterate a distinction here: fully automated AI self-improvement doesn’t at all guarantee an intelligence explosion. Diminishing returns in AI capabilities dampen that feedback loop. Parallel agents can help automate a search across AI parameters, but don’t look nearly as useful in making new conceptual breakthroughs. My expectation is that we’ll need those sorts of breakthroughs for self-improvement to become self-sustaining, let alone to accelerate rapidly and reach truly profound levels of superintelligence. Even a billion parallel AIs also face diminishing returns Today, let’s examine in much more detail what it would mean to run thousands, millions, or billions of AI researchers in parallel. In this post I’m focusing only on software: AI researchers improving the algorithms that make AI better or cheaper to run. Hardware, investment, and economic growth have their own feedback loops, which we discussed briefly in the last post. Here, we’re asking how much more the AI minds themselves can discover. Here’s the argument, with links: - The country of geniuses and the assumptions about adding agents. - The different shapes of diminishing returns. - Where more agents buy speed, and what that speed costs. - Intelligence gains are much harder than speed gains. - The best examples of improving collective intelligence. - What the evidence means for runaway RSI. - Whether scaling is improving. - What we should measure. - The Anthropic test and what would change my mind. A country of geniuses in a datacenter In Machines of Loving Grace, Dario Amodei imagines a: “country of geniuses in a datacenter” Millions of AI minds, each more capable than a Nobel Prize winner across many fields, working at much higher speed. That would be extraordinary. The intelligence of each mind is a crucial part of the premise. Tom Davidson and Daniel Eth’s Forethought paper describes the equivalent of “millions of top-notch human researchers.” Benjamin Todd’s explainer points to “millions of competent engineers working on AI research.” Dwarkesh Patel emphasizes AI’s “massive population size and perfect knowledge transfer.” How does the number of AI agents running in parallel enter these arguments? In Eth and Davidson’s illustrative model, twice as many researchers produce twice as many papers. Cheaper AI lets the same hardware run more researchers, while each software improvement gets harder to find. Alan Chan and colleagues’ 2026 paper, including AI pioneers Geoffrey Hinton and Yoshua Bengio, assumes an even larger effect from adding researchers: doubling effective research labor increases the rate of software improvement by about 2.6 times, holding the software level fixed. Those are assumptions about what more researchers produce. The growing number of AI researchers must keep finding improvements fast enough to outweigh the increasing difficulty of further progress. We’ll come back to those assumptions after looking at what larger AI teams actually achieve. Meanwhile, we had OpenAI’s Navier–Stokes announcement: around 10,000 agents working concurrently during an 88-hour run solved a mathematics Millenium Problem. It made the idea vivid. A datacenter full of AI mathematicians, tackling a famously hard problem. I understand the excitement. But the number of agents tells us how many AI researchers we ran. We also need to measure what those additional researchers contributed. Not all diminishing returns are alike On MTS Live, I described a ladder of increasingly hard scaling: - Super-linear: Twice the input produces more than twice the output. Accelerating returns. - Linear: Twice the input produces twice the output. Independent jobs can look like this. A power-law exponent of 1 is the same case. - Power-law diminishing returns: Output grows as input raised to a power below 1. At exponent 0.5, doubling output takes four times the input. At 0.2, it takes 32 times. At 0.1, it takes 1,024 times. - Logarithmic: Each multiplication of input buys the same additional gain. Another five points might require another tenfold increase in compute. - Sub-logarithmic: Even successive tenfold increases buy smaller gains. Those gains can keep shrinking, or saturate at a ceiling. A slow rise doesn’t require a hard plateau. Not all diminishing returns are alike. Some are much harder than others. The classic pre-training scaling laws describe power-law reductions in prediction loss. Many test-time score curves look roughly logarithmic over the measured range. Some multi-agent results flatten much more sharply. Let’s illustrate this with numbers. In the figure below, every curve earns the same gain from the first doubling of input. The later differences then become visible. The hardest curves share a vertical scale, and each panel states its endpoint. Different diminishing returns scaling patterns lead to very different long term results. For runaway RSI, improvements have to make subsequent improvements arrive faster and faster. Logarithmic returns to the number of AI researchers are too weak to sustain that loop if every further doubling of software efficiency requires a fixed percentage more research. Sub-logarithmic scaling or a plateau are weaker still. The experiments I’ll summarize in the rest of this post aren’t perfect, and I wish we had more data. But the clearest larger-team evidence we have today points to logarithmic or sub-logarithmic intelligence gains. Those look too weak to make RSI self-sustaining, let alone runaway, under the assumptions we’re examining. What limits multiagent scaling There are at least three reasons a larger team might be limited in its ability to produce better answers, and probably more. Let’s start with these: First, coordination skills. Can the agents divide the problem, manage the work, check each other, and combine their results? Can they communicate what matters without spending all their time communicating? A team needs good project management as well as good individual contributors. Second, homogeneity of thought. Are the agents contributing useful new ideas, or repeating the same approaches and mistakes? A hundred copies can share the same blind spots. More varied answers help only if some of that variety is right. Third, the structure of the problem. How much can happen in parallel, and how much has to wait for earlier results? A million independent searches can run at once. A sequential investigation can’t. Even figuring out how to divide a conceptual problem may require an insight the team hasn’t found. Three places where more agents can run into limits. These limits interact. A better model can supply an idea its copies previously lacked, find a better division of labor, or recognize a solution the team had been overlooking. A difficult task may become easier to parallelize once somebody discovers the right approach. That’s why the shape of the curve matters, and why it can change. Before we go further, let’s separate two things: speed, or how quickly agents solve a problem, and collective intelligence, or whether they can solve a harder problem and reach a new level of skill. We’ll use verified benchmark scores as evidence of that intelligence. They aren’t a universal IQ scale. So far, adding agents seems much more effective at speeding up execution than at raising the level of problem a team can solve. So far, using more agents buys us much more speed than extra capability. Agents can buy you speed at a higher cost Let’s start with the best case. A million independent jobs. A million candidates to test. A million places to search. Each agent gets useful work it can do without waiting for the others, and a reliable way to check whether it succeeded. That can scale extremely well. Ten workers may finish ten jobs in the time one worker finishes one. A huge search may find a rare solution that a smaller search misses entirely. One concerning use of parallel agents is offensive cybersecurity. Lisan al-Ghaib emphasizes this possibility. I disagree with his broader argument about swarm intelligence, but many attacks running in parallel could be extremely powerful. A swarm may not need a conceptual breakthrough to search many targets or try many possible vulnerabilities at once. That said, penetrating the most secure targets, such as operating systems and browsers, can require long attack chains. Each step depends on earlier discoveries to get through another layer of defenses. Here again, we might find diminishing returns or eventual saturation, even with respect to speed. Indeed, one of the experiments below suggests that. The speed-cost tradeoff Toby Ord analyzed how well GPT-5.6 Sol increases the speed of solving problems on different benchmarks, and at what cost. This gives us a useful way to measure the tradeoff. In general, using more agents can help get tasks done more quickly, but at higher total cost than if you’d used fewer agents. On top of that, the tradeoff varies by the type of task. On command-line tasks (Terminal-Bench), Ord’s analysis finds that four agents can reach the same result about 1.9 times faster, at about 2.1 times the token cost. On web research (BrowseComp), four agents can reach the same result about 2.6 times faster, at about 1.6 times the cost. I’m using the figure below to show how different tasks differ in that speed vs cost tradeoff. More speed costs more, and the tradeoff varies by task. Based on Toby Ord’s analysis. Source. Why the difference? Web research can often split into independent searches. A command-line task may have to change a shared environment in sequence. Some problems are simply easier to divide among agents than others. Ord uses lambda, λ, for how speed scales with the number of agents. We’ll come back to it. At λ = 0.5, speed and total compute cost rise by the same factor. A higher λ means speed rises faster than cost. Opus 5.5 gets more speed out of parallel agents than older models Inspired by Toby Ord’s analysis, I applied the same approach to Anthropic’s published results on program reconstruction, or ProgramBench. Here again, more speed comes with more cost. But we also see that models within the Claude family are getting better at the speed-cost tradeoff. The newest model, Opus 5.5, gets more speedup for a smaller cost increase than Opus 5 or Fable 5.1. With five agents, Opus 5.5 gets about 2.7 times its solo speed, compared with 2.2 times for Opus 5, and does so at lower cost. [Note: We’re using total agent runtime as a proxy for cost. Token usage and dollar costs could differ.] Opus 5.5 gets more speedup for a smaller cost increase than older AI models. Source. Speed gains probably get harder as we add more agents Will these speed-versus-cost gains hold up at hundreds, thousands, or millions of agents? Probably not completely, and certainly not for all tasks. Some problems are easy to split ten ways, harder to split a hundred ways, and harder still to split among ten thousand agents. More of the work may have to wait for earlier results, and coordinating all those pieces takes work too. Smarter models can get better at this. But my best guess is that, on many problems, each further expansion will buy less speed for the added cost. Anthropic’s formal math proof benchmark gives us one example. On the Lean formal math task, we can compare how quickly teams reach the same score. Using Anthropic’s published checkpoints, we estimate the speed and cost of reaching 0.50. The chart starts with ten agents as the baseline. On a math proof benchmark, the speed advantage of adding additional agents starts to taper with scale. Source. From ten to thirty agents, speed improves faster than cost. From thirty to a hundred, cost rises faster than speed. The local speed exponent, λ, falls from about 0.75 to 0.33. The extra agents still save time, but the tradeoff gets worse. This suggests that even speed gains can taper as we split the work among more agents, let alone gains in intelligence. More agents may add less and less as we approach the limits of how easily a problem divides into subproblems. Better coordination may push those limits outward; the task itself may still impose them. Intelligence gains face much harder diminishing returns than speed So in speed, more agents can buy us time at some additional cost. Newer models are improving, but there are probably limits. What about intelligence? Most arguments for superintelligence go beyond running huge numbers of today’s agents. They posit individual models becoming vastly smarter. There’s a chicken-and-egg problem here. Getting there may require conceptual breakthroughs in machine learning itself, perhaps as profound as the ideas that made modern machine learning possible, or at least as important as the Transformer and reasoning models. Parallel search may not be enough to find them. Anthropic has identified ambitious research ideas as an area where current models lag behind humans. In the previous post, I argued that today’s measured gains from smarter models are roughly five to ten times too weak to produce a self-sustaining feedback loop. But what about running enormous numbers of our best models in parallel? Can a massive team of frontier agents that aren’t yet superintelligent invent a new architecture, a new learning method, or the idea needed to build a much smarter successor? That’s the heart of the question. For runaway RSI, those discoveries have to overcome the diminishing returns we discussed in the last post. Here are three places they appear. The diminishing returns a runaway research loop has to change or overcome. The pre-training example comes from the Chinchilla scaling fit. The reinforcement-learning example comes from ScaleRL. The inference row describes the test-time math curves we discussed previously. The goal is better returns to data and compute, and gains that continue longer before flattening. My expectation is that we’ll get easy AI improvements through work we can automate, parallelize, and search. But to get very large benefits, and runaway RSI, I expect we’ll need conceptual breakthroughs. That’s why the intelligence of each model matters so much. Bear in mind that we’ve observed these sorts of diminishing returns in machine learning for more than thirty years. Power-law learning curves were measured in 1993. They appear to be a fundamental feature of machine learning as we know it. We shouldn’t assume they’ll be trivially changed, or that we’ll suddenly find a way to improve AI without diminishing returns. That would be a revolution. Discovering such a route might itself require far smarter minds, and we don’t know whether the mathematics of learning would permit it across the problems we care about. That doesn’t mean there haven’t been big improvements. The transformer and reasoning models have moved the field forward enormously. But, as we saw in the previous post, a runaway feedback loop needs enough acceleration to keep outweighing the obstacles, again and again. As ideas get harder to find, we may need a faster research pace just to maintain today’s rate of progress. Anthropic’s assessment suggests that better AI tools are becoming necessary just to maintain the current pace of AI progress. Example one: GPT-5.6 cybersecurity gains shrink with more agents OpenAI’s GPT-5.6 Sol results provide a useful example on cybersecurity. SEC-Bench Pro tests the generation of proof-of-concept demonstrations of software vulnerabilities. This is a particular security task, not a measure of coding ability in general. The reported peak scores are approximately 71.2 with one agent, 74.3 with four, and 76.4 with sixteen. On this cybersecurity test, sixteen times as many agents buy about 5.2 score points. The first fourfold increase adds 3.1 points; the next adds only 2.1. There’s still plenty of room below a perfect score, but the gain on this test is shrinking, even with multiplication of the number of agents. This is sub-logarithmic scaling. On SEC-Bench Pro, each expansion buys fewer score points, showing sub-log scaling. Source. These agents could coordinate through OpenAI’s ultra setup. This is already a test of collaboration, not just a stack of independent guesses. The points are the best reported results across different budgets, rather than an equal-compute comparison. Example two: A smarter model did more for OpenAI’s math results than more agents When OpenAI announced its Navier–Stokes result, the 10,000 agents attracted enormous excitement. Interviewing Noam Brown, Dwarkesh Patel said he was “shocked by the scale of cognitive effort” concentrated into such a short run. But how much did those extra agents contribute? Brown, an OpenAI researcher who helped develop o1 and now works on multi-agent systems, was emphatic: “I wouldn’t even attribute 10% of the credit to multi-agent.” He credits the smarter underlying model. OpenAI hadn’t run the single-agent experiment needed to isolate the swarm’s contribution. Brown’s estimate is a judgment, but it directly challenges the idea that the sheer number of agents was the decisive ingredient. Example three: Formal math proof gains diminish sharply after ten Opus 5.5 agents Formal mathematical work seems especially analogous to the conceptual breakthroughs AI research may need. It demands a chain of valid reasoning, sometimes with a new insight that makes the whole problem tractable. Formalizing existing mathematics isn’t the same as inventing a new learning algorithm, but it’s a useful place to look. And here we find almost no extra gain when Opus 5.5 grows from ten parallel agents to a hundred. Anthropic’s system card tests teams on 49 weighted theorem statements, checked in Lean. Each team gets 24 hours. The first tenfold expansion adds 27 score points; the next adds just two. On these math proofs, most of the gain arrives by ten agents. This is steeply sub-logarithmic scaling, and possibly saturation Source. These agents could communicate, share a machine and Git repository, and organize their work. The hundred-agent team even developed a hierarchy with twelve sub-leads. It could also use ten times as much aggregate agent runtime as the ten-agent team over the same day. That isn’t a measured tenfold increase in tokens, but it gives the larger team much more room to work. We still didn’t get over the hump. This won’t necessarily stay true, but the diminishing returns are incredibly steep here, possibly pointing toward a plateau. What the formal math proof slowdown suggests Opus 5.5 starts smarter than the older models and gets more speedup from parallel agents on ProgramBench. It clearly can put additional agents to useful work. So why do intelligence gains nearly stop on these math proofs? Two possibilities stand out: - The remaining work may be hard to parallelize. Later steps depend on understanding earlier results. - The remaining problems may need insights the model can’t find, however many copies we run. Intelligence may matter more than agent population or speed. We haven’t isolated the cause. But a thousand copies don’t automatically supply the insight one copy is missing. Example four: A smarter model beats ten times the agents on math proofs Looking at the same math results, we see something striking. Ten Opus 5.5 agents get a higher score than a hundred Fable 5.1 or Opus 5.0 agents. Opus 5.5 starts much higher than the older models. It improves substantially with ten agents, then barely improves with a hundred. But at 10 it’s already better than Fable or Opus 5’s results with a hundred agents. Opus 5.5 starts stronger, but its team gains flatten quickly. Source. Ten Opus 5.5 agents beat a hundred Opus 5 agents on these math proofs. They beat a hundred Fable 5.1 agents too. The stronger model does more with one-tenth as many agents. Ten Opus 5.5 agents are able to solve problems a hundred older-model agents can’t. Source. We saw an analogous pattern with test-time compute in the previous post. OpenAI’s internal model starts far above Astra on the curated math problems below. Even its lowest displayed result beats Astra’s highest. Changing the model gives a much bigger boost than spending more compute on Astra over the tested range. A stronger model makes the bigger difference than more test-time compute in solving previously-unsolved math problems. Source. A rough extrapolation of Astra’s line suggests it would need about 300 times as much compute as the internal model to match the internal model’s lowest plotted score. And that extrapolation itself is an unknown. It’s quite possible that Astra would just plateau before ever matching the OpenAI internal model’s results with far less compute. What about diversity? Does adding different minds help? A study of model and persona diversity found substantial gains as diverse teams grew to eight agents. After that, the average across its tasks barely moved. Diversity of AI models helps, but the gains flatten after eight to twelve agents. Source. Other approaches, including X-MAS and Aitchison and colleagues’ forecasting study, find useful gains from combining different models. But none of these results demonstrates sustained improvement over many expansions of the team. The first study directly shows flattening; the others test smaller ranges. It’s plausible that today’s models simply aren’t diverse enough. LLMs often produce strikingly similar ideas. Perhaps we need bigger changes in how these minds are trained. We’ll return to whether more deliberately different agents could change the curves. The Best Examples of Multi Agents Improving Collective Intelligence All of the above examples show sub-logarithmic scaling, roughly the hardest kind. That said, some experiments show slightly easier diminishing returns, indicating roughly the same gain from each multiplication of parallel work. That’s logarithmic scaling. It’s better, but still steep diminishing returns. If that’s how useful research grows while each further AI improvement gets harder to find, even billions of agents don’t make a self-sustaining RSI loop look plausible. In Ryan Greenblatt’s ARC experiment, a model generated and checked many candidate programs. More samples kept improving its score, by roughly the same amount per doubling. This is parallel search rather than a communicating research team, but it gives us a useful comparison. Parallel search keeps improving, with logarithmic returns. Source. A molecular-exploration study found a similar logarithmic pattern as it grew from one to 128 agents. Multiple agents explored chemical targets more effectively than one agent given the same total evaluation budget. Each agent learned as it searched. More AI agents explore more chemical targets, with logarithmic diminishing returns. Source. There are encouraging results from communicating teams too, including Argus on web research and Microsoft and Berkeley’s work on ARC-AGI-3. The latter finds unusually strong gains from one to five agents. Those results deserve attention; we still need to see what happens with much larger teams. Logarithmic gains are better than the sub-logarithmic curves we saw earlier. Each tenfold expansion keeps buying the same extra increment, instead of less each time. But both shapes rise drastically more slowly than the research gains assumed in hypothetical models of self-sustaining or runaway RSI. Let’s put them side by side. Which problems should scale best Some work can be split into independent searches or experiments. Other work needs the previous result before it can proceed. And some problems may need a higher level of intelligence or a conceptual breakthrough before we even know how to divide them. Here’s a very tentative and relatively subjective stab at which problems will scale best and worse with multiple agents. The chart runs from work that seems easiest to parallelize at the top to the hardest at the bottom. I’m sure others have many thoughts here, and that this can be much improved upon. From work that seems easiest to parallelize to the hardest. What does this mean for runaway RSI Will a million or a billion parallel AIs get us to self-sustaining or runaway RSI? Let’s do the math. Hypothetical software-RSI models assume that a growing number of AI researchers produces a much larger stream of discoveries. Those discoveries make AI cheaper or more productive, allowing the next round of research to grow again. There’s a strong case for gains in speed and for searching more of a well-defined space. Even speed gains, as we’ve seen, can taper. Intelligence is much harder to boost through parallel agents. In several experiments, peak problem-solving performance (what I’m referring to here as intelligence) rises dramatically more slowly than speed. Here’s what three software-RSI models assume: - Chan and colleagues (2026): doubling effective research labor produces about 2.6 times the rate of software improvement, holding the software level fixed. I find that implausible as a general assumption about adding copies of to
Ramez Naam is, in my opinion, one of the best futurists alive. But although he has a Substack (which you should definitely follow), he doesn’t write nearly enough. So I’ve been encouraging him to write more by cross-posting his posts on my blog. Somewhat to my surprise, Ramez has turned out to be a lot more of an AI skeptic than I would have though. He’s certainly not dismissive of the technology, but he disagrees with many of the more expansive and dramatic claims coming out of the big labs (especially Anthropic). In his first post in this series, he expressed skepticism that recursive self-improvement (RSI) would lead to a fast takeoff of AI capabilities in the near future: In this post, he expresses skepticism about the potential for agent swarms — large numbers of AI agents working in parallel — to offer dramatic increases in AI capabilities. Agent swarms have been in the news a lot, both because of the Hugging Face attack, and OpenAI’s solution of the famous Navier-Stokes problem is also widely attributed to a large swarm. But Ramez argues that parallelization is just another kind of input that suffers from diminishing returns, and that we should therefore temper our expectations about what agents swarms are able to achieve. As was the case with his previous post, I’m not sure how much this matters, when the capabilities of the underlying models — swarm or no swarm — are still improving at such a steady and impressive rate. But it’s a very intellectually interesting argument, and Ramez can be very persuasive. In my previous post, we asked whether AIs could improve themselves all the way to absurdly high levels of superintelligence. My conclusion: today’s feedback loop is probably too weak, because improving AI faces diminishing returns at every step. Even fully autonomous AIs doing all the research and coding don’t make those diminishing returns disappear. We should keep updating that conclusion as real-world parameters and data change. But we didn’t look closely at another possibility: running millions or billions of AI minds in parallel. How much do they help? That’s what we’ll look at here. TL;DR Plenty of people have modeled how more AI agents working in parallel could fuel recursive self-improvement. But the data we have today tell a different story. More agents are useful for speed: they can solve many problems faster. They’re less useful for boosting AI intelligence. Multi-agent gains in solving harder problems often diminish incredibly quickly and may even saturate. Whether multi-agent systems can produce the conceptual breakthroughs RSI might require to become self-sustaining is the harder question. Without breakthroughs in how well agents parallelize problems - something that may depend on the structure of machine learning problems themselves - it’s hard to see even millions or billions of copies of today’s models making repeated breakthroughs on the hardest problems. These are among the strongest diminishing returns we’ve looked at. Of course, these diminishing returns may change. And we have less experimental data to go off of today than we’d like. We’ll need to recalibrate as new evidence comes in. I’ll end with the things we should track and my predictions about a natural experiment from newly released Anthropic data. Below is the two-image summary: The experimental observations of how large numbers of agents scale to solve harder problems (what I’m referring to as intelligence) are far far lower than the assumptions in studies that predict that AI recursive self-improvement will lead to an intelligence explosion. The first image is at the scale of 100 parallel AI agents. The second is at the scale of a billion AI parallel agents. Real world experiments show dramatically lower intelligence gains from parallel agents than hypothetical AI models. Extrapolated out to a billion agents, the intelligence gain in hypothetical models is orders of magnitude beyond real world experiment gain curves. What we’ve already covered regarding RSI and diminishing returns To recap the last post: recursive self-improvement (RSI) means AI improvements feeding back into the next round of AI research. Four findings stood out: - Real AI research is harder than benchmark scores suggest. OpenAI’s models needed human help on much shorter research tasks than the coding benchmarks led us to expect. - More AI activity produces much smaller gains in useful work. OpenAI used 124 times as many tokens per researcher but ran only 1.6 times as many experiments per researcher. - Maintaining progress requires ever more resources. Training and thinking time face steep diminishing returns. Anthropic reports that more powerful AI has been key to maintaining the current pace of AI progress. - The measured software feedback loop doesn’t yet look self-sustaining. Our tentative calibration put it roughly five to ten times below the strength needed. RSI doesn’t necessarily mean runaway superintelligence I want to reiterate a distinction here: fully automated AI self-improvement doesn’t at all guarantee an intelligence explosion. Diminishing returns in AI capabilities dampen that feedback loop. Parallel agents can help automate a search across AI parameters, but don’t look nearly as useful in making new conceptual breakthroughs. My expectation is that we’ll need those sorts of breakthroughs for self-improvement to become self-sustaining, let alone to accelerate rapidly and reach truly profound levels of superintelligence. Even a billion parallel AIs also face diminishing returns Today, let’s examine in much more detail what it would mean to run thousands, millions, or billions of AI researchers in parallel. In this post I’m focusing only on software: AI researchers improving the algorithms that make AI better or cheaper to run. Hardware, investment, and economic growth have their own feedback loops, which we discussed briefly in the last post. Here, we’re asking how much more the AI minds themselves can discover. Here’s the argument, with links: - The country of geniuses and the assumptions about adding agents. - The different shapes of diminishing returns. - Where more agents buy speed, and what that speed costs. - Intelligence gains are much harder than speed gains. - The best examples of improving collective intelligence. - What the evidence means for runaway RSI. - Whether scaling is improving. - What we should measure. - The Anthropic test and what would change my mind. A country of geniuses in a datacenter In Machines of Loving Grace, Dario Amodei imagines a: “country of geniuses in a datacenter” Millions of AI minds, each more capable than a Nobel Prize winner across many fields, working at much higher speed. That would be extraordinary. The intelligence of each mind is a crucial part of the premise. Tom Davidson and Daniel Eth’s Forethought paper describes the equivalent of “millions of top-notch human researchers.” Benjamin Todd’s explainer points to “millions of competent engineers working on AI research.” Dwarkesh Patel emphasizes AI’s “massive population size and perfect knowledge transfer.” How does the number of AI agents running in parallel enter these arguments? In Eth and Davidson’s illustrative model, twice as many researchers produce twice as many papers. Cheaper AI lets the same hardware run more researchers, while each software improvement gets harder to find. Alan Chan and colleagues’ 2026 paper, including AI pioneers Geoffrey Hinton and Yoshua Bengio, assumes an even larger effect from adding researchers: doubling effective research labor increases the rate of software improvement by about 2.6 times, holding the software level fixed. Those are assumptions about what more researchers produce. The growing number of AI researchers must keep finding improvements fast enough to outweigh the increasing difficulty of further progress. We’ll come back to those assumptions after looking at what larger AI teams actually achieve. Meanwhile, we had OpenAI’s Navier–Stokes announcement: around 10,000 agents working concurrently during an 88-hour run solved a mathematics Millenium Problem. It made the idea vivid. A datacenter full of AI mathematicians, tackling a famously hard problem. I understand the excitement. But the number of agents tells us how many AI researchers we ran. We also need to measure what those additional researchers contributed. Not all diminishing returns are alike On MTS Live, I described a ladder of increasingly hard scaling: - Super-linear: Twice the input produces more than twice the output. Accelerating returns. - Linear: Twice the input produces twice the output. Independent jobs can look like this. A power-law exponent of 1 is the same case. - Power-law diminishing returns: Output grows as input raised to a power below 1. At exponent 0.5, doubling output takes four times the input. At 0.2, it takes 32 times. At 0.1, it takes 1,024 times. - Logarithmic: Each multiplication of input buys the same additional gain. Another five points might require another tenfold increase in compute. - Sub-logarithmic: Even successive tenfold increases buy smaller gains. Those gains can keep shrinking, or saturate at a ceiling. A slow rise doesn’t require a hard plateau. Not all diminishing returns are alike. Some are much harder than others. The classic pre-training scaling laws describe power-law reductions in prediction loss. Many test-time score curves look roughly logarithmic over the measured range. Some multi-agent results flatten much more sharply. Let’s illustrate this with numbers. In the figure below, every curve earns the same gain from the first doubling of input. The later differences then become visible. The hardest curves share a vertical scale, and each panel states its endpoint. Different diminishing returns scaling patterns lead to very different long term results. For runaway RSI, improvements have to make subsequent improvements arrive faster and faster. Logarithmic returns to the number of AI researchers are too weak to sustain that loop if every further doubling of software efficiency requires a fixed percentage more research. Sub-logarithmic scaling or a plateau are weaker still. The experiments I’ll summarize in the rest of this post aren’t perfect, and I wish we had more data. But the clearest larger-team evidence we have today points to logarithmic or sub-logarithmic intelligence gains. Those look too weak to make RSI self-sustaining, let alone runaway, under the assumptions we’re examining. What limits multiagent scaling There are at least three reasons a larger team might be limited in its ability to produce better answers, and probably more. Let’s start with these: First, coordination skills. Can the agents divide the problem, manage the work, check each other, and combine their results? Can they communicate what matters without spending all their time communicating? A team needs good project management as well as good individual contributors. Second, homogeneity of thought. Are the agents contributing useful new ideas, or repeating the same approaches and mistakes? A hundred copies can share the same blind spots. More varied answers help only if some of that variety is right. Third, the structure of the problem. How much can happen in parallel, and how much has to wait for earlier results? A million independent searches can run at once. A sequential investigation can’t. Even figuring out how to divide a conceptual problem may require an insight the team hasn’t found. Three places where more agents can run into limits. These limits interact. A better model can supply an idea its copies previously lacked, find a better division of labor, or recognize a solution the team had been overlooking. A difficult task may become easier to parallelize once somebody discovers the right approach. That’s why the shape of the curve matters, and why it can change. Before we go further, let’s separate two things: speed, or how quickly agents solve a problem, and collect
document fetched 2026-10-24 11:26 · canonical https://noahpinion.blog/p/can-a-million-ai-agents-create-superintelligence