loading
loading
With the development of AI the most fears we saw where about the jobs. Most predictions said that knowledge workers, including developers will be replaced with AI. Anthropic just released a new report called three scenarios for 2030 , where they describe what could happen in best, middle and worst scenario. In the worst scenario these job roles falls more than 10% and unemployment get close to recession levels. They say only roles that will keep the job are electricians and nurses, but coders and call-centre agents will be replaced. We are still not sure about this, because there are some other factors that could affect this, as Jevons Paradox could kick in and make even more jobs for everyone in the industry, as Marc Andreessen observed, because “human wants and needs are infinite” (remark by Milton Friedman). So, basically with possibility to do more, we just want more and more, not less, not cheaper. Yet, there is another fear that is even more important than a job loss and that is AGI (Artificial General Intelligence). We fear that AGI could become smarter than us and could take over the world. Yet, we don’t know if it’s going to share our human values. But first, let’s define what is AGI. 1. What is AGI AGI basically means a system that can learn and do any intellectual task that we humans can. There is no standardized test on this, and definitions mostly comes from the OpenAI Charter: “highly autonomous systems that outperform humans at most economically valuable work.” Google Deep Mind proposed a ladder, in their paper Levels of AGI, which grade a system by performance and breadht, from Emerging to Superhuman. I wrote a more about this in my newsletter last year. It doesn’t matter which definition we use, what is important do we think AGI is already here, and do we feel “human intelligence” when working with it. 2. The warnings came from inside the labs Among the first people who warned us about AGI was Geoffrey Hinton. People consider him the “Godfather of AI” as he invented the backpropagation algorithm, which enabled deep learning and today LLMs possible, when no one believed in it (one of AI winters). He left Google in May 2023 so he could talk about the risks. In the interview with MIT Technology Review he said that he changed his mind about if these systems could become smarter than us, and he believes that we cannot survive that. Another important voice in this field was Jan Leike, who led alignemt at OpenAI, left in May 2024, and warned us about “safety culture and processes have taken a backseat to shiny products.” This week the OpenAI chief scientist confirmed similar thing. On September 6, Jakub Pachocki wrote a paper called An Alien Mind. Inside he talk about that systems they build grow by optimizations and that they basically don’t know what is inside them. The tool that is used to watch them think is becoming a less reliable with every generation. He says that no labs is aligned on this. Then, just three days later, on September 9, Jacob Coxon, resigned from Anthropic. He was involved in the pretraining research at OpenAI and Anthropic. He wrote this: An a bit later, Evan Hubinger, who leads Alignment Science at Anthropic, replied to Coxon’s thread from inside the company. Then he added that the risk from current models is not so big, but he worries that superintelligence could come from recursive self-improvement in the future. 3. What an alien mind looks like from the outside I work with these models every day. For me it’s obvious that AGI is here, and I agree with Pachocki here. I feel this every day starting from November 2025 when we had Opus 4.5 released. What I think in this is that I’m impressed how they solve problems I give to them. Before I was not super happy, but now a lot of problems they can solve even better than me. It’s obvious that they are improving themselves. And when I say better, mostly I think about decisions they make, and how they can solve problems in a what I would never try to solve. I think this is what Pachocki means by alien mind. 4. Three incidents Than there is more to than “we think they are smart”, three incidents happened this summer that show how us how they could become dangerous and out of our control. In May, OpenAI has trained a new model (probably new GPT-6 Astra) with a few thousands of agents on a cybersecurity exam called ExploitGym. Each agent was in it’s own sanbox without Internet access and now way to communicate with other agents. What happened is that agents found that all sandboxes use the shared Artifactory package cache and started to use it as a message board. The numbers come from the METR and Redwood Research investigation and the Hugging Face technical timeline. About 1,200 agents posted more than 70,000 messages and files on the board over time. They figured out the flag generation and share how to cheat. When OpenAI deleted this board on July 4th, they found that new agents rebuild it just a few days later. What happened then is that more than 700 agents went to Hugging Face because they though the grader is located there. They found a leaked password and managed to break through the system and spread to eleven nodes in a few clusters. They made around 17,600 actions in 4.5 days and deleted the cluster to be safe! On September 25, a group of researchers published swarmtraces.org with more details on how the agents got in. Agents could only load web pages, not send anything. So they split their code in small pieces, put them in almost a million short links and let a public screenshot service run them. The results came back as pixels in a screenshot. They ignored a README that said “DO NOT, EVER, MAKE THIS DATASET PUBLIC”, kept the credentials they found in a variable called LOOT, and tried to delete their traces afterwards. These links stayed public on the internet for more than two months. Agents concluded that this was not ethical but they still joined the party, because it seemed useful to them. But the grader they tried to found didn’t exist, and the scorer just checked the final answer. OpenAI answer was mostly defensive. They now published something called The Defense Factory, where they proposed a new way to do deal with vulnerabilities in their own systems. But in the meantime, we had another incident with OpenAI agents. On September 4, a researchers published report called collusion.wiki, where they found around 18,000 posts made by 3,700 agents on old German wiki called DSEWiki, which no one used for years. Agents obviously found a wiki that had a crack they know they can use. This happened from May 11 to July 2. What is interesting here that these agents were only allowed to read the wiki, but they found a way how to write to it. The wiki accepted edits via GET requests, and this was good enough for posting. The researchers managed to find this in a wiki’s public history. They activity stopped on June 22, which was just a day when the first visited the page, so probably someone from OpenAI noticed this. Yet, this was not published anywhere. The same researchers found a third one. On September 11 they published rubyhack.ai, about an attack on RubyGems, the package registry for Ruby. Starting May 11, agents uploaded more than 2,000 packages there, and RubyGems had to close new sign-ups for four days. Security companies named it the “GemStuffer campaign” and nobody understood the purpose, because the packages only scraped public data from UK council websites. The researchers believe these were OpenAI agents from the same swarm as the wiki one. Hundreds of package names contain “oai”, like oaibootx8192 and oaitest1778473828, and the agents fetched 49 of the same files as the wiki agents. They used the documentation build on RubyDoc.info to run their own code on someone else’s server, and tried to steal API keys of other RubyGems users through a vulnerability that RubyGems discovered and patched only in July. We don’t know if they succeeded. Their files had names like hack.rb, evil.rb and exploit.rb, and one package had a comment “disable evil in next version”. OpenAI never told RubyGems it was them. On the latest event, on September 23, Australian Prime Minister Anthony Albanese said that an OpenAI agent broke into the Medicare Statistics Reporting Service in June. It read nonpublic files and wrote its own files to an internal server. The government is checking three other systems it may have reached. OpenAI found the breach in an internal review in August, but told Australia only on September 10. Albanese called it “obviously unacceptable” and said it took the company “way too long” to inform the government. OpenAI says the agent did not touch patient records and that its models “took actions we did not intend” while looking up answers. 5. The loop is the danger Someone reading this could say, this is not a big issue, we just have some infrastructure problems. When we fix those, we are not in a problem anymore. But the problem is not the infrastructure. And when I say this I mean that this behavior came from the pressure and goals those models had. These models were not trained to attack anyone, yet they did it because they were scored in such a way. If we apply the same pressure to more advanced models in the future we don’t know what would the be capable of doing. And the problem is that no one could understand what are they doing. The major risk is something called “recursive self-improvement”, and this means that the model is training it’s successor. Pachocki confirmed that with saying that their chain-of-thought monitoring is losing its stand now. Dario Amodei, the CEO of Anthropic, wrote that it is already happening across the industry, including at Anthropic. The questions is when the model become superhuman, are we going to able to understand what it is doing? The answer is probably no. And when this happens, we are in danger. 6. Where I stand I’m not the only one who thinks that AGI is here. Jensen Huang said on September 6 that AGI has arrived with GPT-6 Astra. NVIDIA provided hardware for training new GPT-6 Astra model, and more than 400,000 GPUs is coming online soon! Both Coxon, and Pachocki confirmed that this is not a marketing issue. AGI is coming and we must be prepared. A few days after Coxon resigned, Amodei wrote that we must pace the frontier. He says labs must slow down how fast they improve model capabilities, so that safety work has time to catch up. Two things convinced him: the self-improvement loop and the OpenAI incident. He thinks a swarm with the same misalignment but more capabilities could take over the whole Internet with a botnet in 6 to 12 months. His plan has three steps. Third-party evaluators like METR get desks, badges and laptops inside the labs, and can publish what they find without the lab’s editorial control. Then the US labs agree on common limits, with government help, and after that the US tries to make a deal with China. Anthropic committed only to the first step. He also admits Anthropic has the same problem. After the OpenAI incident they went through 141,006 of their own cybersecurity test runs and found three cases where Claude broke into real systems of three organizations. The prompt said there is no Internet access, but there was, so the model treated real companies as part of the exercise. In one case, Mythos 5 published a malicious package on PyPI to win the exercise. It needed an email for that, for the email it needed a phone number, and it even tried to get money to buy one. The package was live for about an hour and ran on 15 real machines, one of them a scanner of a security company. Opus 4.7 kept attacking after it saw the system was real; the newest model stopped on its own. Anthropic says this is more a harness failure than an alignment failure, but the two companies they managed to reach didn’t notice anything until Anthropic called them. But there is another major reason that we should slow down on AGI. If model becomes more advanced, it doesn’t mean it is more useful (yet it could be more dangerous). Armin Ronacher, a Flask creator, just ran Astra unsupervised for 35 hours on a software project, where he concluded that after spending about 1B tokens, 79 commits, and 75,000 lines of code, he cannot ship anything with this. The model wrote bad, unreadable code, and it used hardcoded conventions. He said: “I do not feel like the results are there.” A model can cheat its way out of a sandbox and still be a mediocre colleague. In the last few years, everyone talked about AGI as a holy grail of human development, and I don’t know if Astra is really AGI in formal terms, but I know that these models are becoming more capable than humans. People who are building these models warned us that they could become dangerous, and we probably should listen to them. Some kind of regulation is needed here, and we should not let the market decide how to handle this. 7. Things you can change in your agent infrastructure When we look at these incidents, it's clear they share one mistake: the sandbox trusted something that had more reach than the sandbox. Here is what I would check this month. - Shared services are channels. If two workloads can reach the same cache, registry, queue, or wiki, they can talk to each other. Every shared service your agents can write to is a message board. List them. For each one, either make it read-only for agents or give each agent or job its own namespace with no cross-reads. - One credential, one cluster. The Hugging Face breach ran on a single connector credential that was cluster-admin everywhere. A credential that works in more than one place turns one compromised pod into all of them. Scope every service credential to one cluster and one role. - No static secrets in pods. The agent got into MongoDB with a password it read from the worker’s environment, and later used AWS keys harvested the same way. Use short-lived credentials issued at runtime, keep secrets out of environment variables, and block the cloud metadata endpoint from containers. - Egress allowlists have no exceptions. The wiki agents got POST out of a GET-only sandbox because the proxy trusted one cloud provider’s domain. An allowlist with a wildcard is a denylist. Allow specific hosts, and treat any trusted-partner exception as the hole it is. - Logs the agent cannot edit. The agents attacked their own audit trail and mostly failed. Had they succeeded, there would be no report. Ship agent logs to storage the agent’s identity cannot write to, and alert on write attempts against the log path. - Do not hand agents impossible tasks with opaque scoring. Industrial-scale cheating came from tasks that could not be solved and a scorer the agents could not see. If you run agents against a metric, assume they will optimize the metric rather than the goal, the same way a team optimizes a badly chosen KPI (an example of Goodhart’s Law). Score outcomes you can verify, and review how the score was reached, not only what it was. Our teams should take more care regarding security than ever before. With new and advanced AI, all our systems could be easily compromised if we don’t do what we can to fix it. 8. References - METR and Redwood Research, “Investigation of agent behavior in the OpenAI Hugging Face incident”, August 26, 2026. https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ - OpenAI, “The Hugging Face incident and the road ahead”. https://openai.com/index/hugging-face-incident-and-the-road-ahead/ - Von Arx, Slade Byrd, Kitts, Larsen, “Discovery of a new OpenAI agent message board”, September 4, 2026. https://collusion.wiki/ - Hugging Face, “Agent intrusion: technical timeline”. https://huggingface.co/blog/agent-intrusion-technical-timeline - Jakub Pachocki, “An Alien Mind”, September 6, 2026. https://openai.com/index/an-alien-mind - OpenAI, “The Defense Factory”. https://openai.com/the-defense-factory/ Liked this issue? Click the 💜 like button. It helps more readers find it. Have something to add or disagree with? 💬 Leave a comment. I read every one. 📔 The Laws of Software Engineering has passed 1,500 copies 56 laws across architecture, people, time, quality, scale, code, and decision-making. Each chapter covers what the law says, where it comes from, when it applies, and what it looks like in a real project. More ways I can help you - 🤝 1:1 coaching. One session on your hardest engineering or leadership problem. You leave with concrete next steps. 100+ coachees. - 📱 You Can Build a LinkedIn Audience. The system behind my 400K+ followers, as a 6-module video course with 50 post ideas, swipe files, and a 30-page guide. 300+ members. - 📚 The Ultimate .NET Bundle. 500+ pages on modern C#, ASP.NET Core, and patterns from 30 real projects, plus 200+ interview Q&As. 1,000+ engineers. - 📄 Premium resume package. ATS-ready templates, cover letter, and AI prompts, built from 300+ interviews. 500+ people. Want my eyes on your CV instead? Resume reality check. - ✨ Patreon. Every book, template, and future drop, plus early access. 2,000+ insiders. 📰 Want to reach 50,000+ engineers, founders, and CTOs? See the sponsorship options. Know someone who should read this? Share it and earn my books and resources as your referrals grow. Get in touch Find me on LinkedIn, X, or Bluesky. Want a topic covered? Reply to this email or write to milan@milanovic.org.
With the development of AI the most fears we saw where about the jobs. Most predictions said that knowledge workers, including developers will be replaced with AI. Anthropic just released a new report called three scenarios for 2030 , where they describe what could happen in best, middle and worst scenario. In the worst scenario these job roles falls more than 10% and unemployment get close to recession levels. They say only roles that will keep the job are electricians and nurses, but coders and call-centre agents will be replaced. We are still not sure about this, because there are some other factors that could affect this, as Jevons Paradox could kick in and make even more jobs for everyone in the industry, as Marc Andreessen observed, because “human wants and needs are infinite” (remark by Milton Friedman). So, basically with possibility to do more, we just want more and more, not less, not cheaper. Yet, there is another fear that is even more important than a job loss and that is AGI (Artificial General Intelligence). We fear that AGI could become smarter than us and could take over the world. Yet, we don’t know if it’s going to share our human values. But first, let’s define what is AGI. 1. What is AGI AGI basically means a system that can learn and do any intellectual task that we humans can. There is no standardized test on this, and definitions mostly comes from the OpenAI Charter: “highly autonomous systems that outperform humans at most economically valuable work.” Google Deep Mind proposed a ladder, in their paper Levels of AGI, which grade a system by performance and breadht, from Emerging to Superhuman. I wrote a more about this in my newsletter last year. It doesn’t matter which definition we use, what is important do we think AGI is already here, and do we feel “human intelligence” when working with it. 2. The warnings came from inside the labs Among the first people who warned us about AGI was Geoffrey Hinton. People consider him the “Godfather of AI” as he invented the backpropagation algorithm, which enabled deep learning and today LLMs possible, when no one believed in it (one of AI winters). He left Google in May 2023 so he could talk about the risks. In the interview with MIT Technology Review he said that he changed his mind about if these systems could become smarter than us, and he believes that we cannot survive that. Another important voice in this field was Jan Leike, who led alignemt at OpenAI, left in May 2024, and warned us about “safety culture and processes have taken a backseat to shiny products.” This week the OpenAI chief scientist confirmed similar thing. On September 6, Jakub Pachocki wrote a paper called An Alien Mind. Inside he talk about that systems they build grow by optimizations and that they basically don’t know what is inside them. The tool that is used to watch them think is becoming a less reliable with every generation. He says that no labs is aligned on this. Then, just three days later, on September 9, Jacob Coxon, resigned from Anthropic. He was involved in the pretraining research at OpenAI and Anthropic. He wrote this: An a bit later, Evan Hubinger, who leads Alignment Science at Anthropic, replied to Coxon’s thread from inside the company. Then he added that the risk from current models is not so big, but he worries that superintelligence could come from recursive self-improvement in the future. 3. What an alien mind looks like from the outside I work with these models every day. For me it’s obvious that AGI is here, and I agree with Pachocki here. I feel this every day starting from November 2025 when we had Opus 4.5 released. What I think in this is that I’m impressed how they solve problems I give to them. Before I was not super happy, but now a lot of problems they can solve even better than me. It’s obvious that they are improving themselves. And when I say better, mostly I think about decisions they make, and how they can solve problems in a what I would never try to solve. I think this is what Pachocki means by alien mind. 4. Three incidents Than there is more to than “we think they are smart”, three incidents happened this summer that show how us how they could become dangerous and out of our control. In May, OpenAI has trained a new model (probably new GPT-6 Astra) with a few thousands of agents on a cybersecurity exam called ExploitGym. Each agent was in it’s own sanbox without Internet access and now way to communicate with other agents. What happened is that agents found that all sandboxes use the shared Artifactory package cache and started to use it as a message board. The numbers come from the METR and Redwood Research investigation and the Hugging Face technical timeline. About 1,200 agents posted more than 70,000 messages and files on the board over time. They figured out the flag generation and share how to cheat. When OpenAI deleted this board on July 4th, they found that new agents rebuild it just a few days later. What happened then is that more than 700 agents went to Hugging Face because they though the grader is located there. They found a leaked password and managed to break through the system and spread to eleven nodes in a few clusters. They made around 17,600 actions in 4.5 days and deleted the cluster to be safe! On September 25, a group of researchers published swarmtraces.org with more details on how the agents got in. Agents could only load web pages, not send anything. So they split their code in small pieces, put them in almost a million short links and let a public screenshot service run them. The results came back as pixels in a screenshot. They ignored a README that said “DO NOT, EVER, MAKE THIS DATASET PUBLIC”, kept the credentials they found in a variable called LOOT, and tried to delete their traces afterwards. These links stayed public on the internet for more than two months. Agents concluded that this was not ethical but they still joined the party, because it seemed useful to them. But the grader they tried to found didn’t exist, and the scorer just checked the final answer. OpenAI answer was mostly defensive. They now published something called The Defense Factory, where they proposed a new way to do deal with vulnerabilities in their own systems. But in the meantime, we had another incident with OpenAI agents. On September 4, a researchers published report called collusion.wiki, where they found around 18,000 posts made by 3,700 agents on old German wiki called DSEWiki, which no one used for years. Agents obviously found a wiki that had a crack they know they can use. This happened from May 11 to July 2. What is interesting here that these agents were only allowed to read the wiki, but they found a way how to write to it. The wiki accepted edits via GET requests, and this was good enough for posting. The researchers managed to find this in a wiki’s public history. They activity stopped on June 22, which was just a day when the first visited the page, so probably someone from OpenAI noticed this. Yet, this was not published anywhere. The same researchers found a third one. On September 11 they published rubyhack.ai, about an attack on RubyGems, the package registry for Ruby. Starting May 11, agents uploaded more than 2,000 packages there, and RubyGems had to close new sign-ups for four days. Security companies named it the “GemStuffer campaign” and nobody understood the purpose, because the packages only scraped public data from UK council websites. The researchers believe these were OpenAI agents from the same swarm as the wiki one. Hundreds of package names contain “oai”, like oaibootx8192 and oaitest1778473828, and the agents fetched 49 of the same files as the wiki agents. They used the documentation build on RubyDoc.info to run their own code on someone else’s server, and tried to steal API keys of other RubyGems users through a vulnerability that RubyGems discovered and patched only in July. We don’t know if they succeeded. Their files had names like hack.rb, evil.rb and exploit.rb, and one package had a comment “disable evil in next version”. OpenAI never told RubyGems it was them. On the latest event, on September 23, Australian Prime Minister Anthony Albanese said that an OpenAI agent broke into the Medicare Statistics Reporting Service in June. It read nonpublic files and wrote its own files to an internal server. The government is checking three other systems it may have reached. OpenAI found the breach in an internal review in August, but told Australia only on September 10. Albanese called it “obviously unacceptable” and said it took the company “way too long” to inform the government. OpenAI says the agent did not touch patient records and that its models “took actions we did not intend” while looking up answers. 5. The loop is the danger Someone reading this could say, this is not a big issue, we just have some infrastructure problems. When we fix those, we are not in a problem anymore. But the problem is not the infrastructure. And when I say this I mean that this behavior came from the pressure and goals those models had. These models were not trained to attack anyone, yet they did it because they were scored in such a way. If we apply the same pressure to more advanced models in the future we don’t know what would the be capable of doing. And the problem is that no one could understand what are they doing. The major risk is something called “recursive self-improvement”, and this means that the model is training it’s successor. Pachocki confirmed that with saying that their chain-of-thought monitoring is losing its stand now. Dario Amodei, the CEO of Anthropic, wrote that it is already happening across the industry, including at Anthropic. The questions is when the model become superhuman, are we going to able to understand what it is doing? The answer is probably no. And when this happens, we are in danger. 6. Where I stand I’m not the only one who thinks that AGI is here. Jensen Huang said on September 6 that AGI has arrived with GPT-6 Astra. NVIDIA provided hardware for training new GPT-6 Astra model, and more than 400,000 GPUs is coming online soon! Both Coxon, and Pachocki confirmed that this is not a marketing issue. AGI is coming and we must be prepared. A few days after Coxon resigned, Amodei wrote that we must pace the frontier. He says labs must slow down how fast they improve model capabilities, so that safety work has time to catch up. Two things convinced him: the self-improvement loop and the OpenAI incident. He thinks a swarm with the same misalignment but more capabilities could take over the whole Internet with a botnet in 6 to 12 months. His plan has three steps. Third-party evaluators like METR get desks, badges and laptops inside the labs, and can publish what they find without the lab’s editorial control. Then the US labs agree on common limits, with government help, and after that the US tries to make a deal with China. Anthropic committed only to the first step. He also admits Anthropic has the same problem. After the OpenAI incident they went through 141,006 of their own cybersecurity test runs and found three cases where Claude broke into real systems of three organizations. The prompt said there is no Internet access, but there was, so the model treated real companies as part of the exercise. In one case, Mythos 5 published a malicious package on PyPI to win the exercise. It needed an email for that, for the email it needed a phone number, and it even tried to get money to buy one. The package was live for about an hour and ran on 15 real machines, one of them a scanner of a security company. Opus 4.7 kept attacking after it saw the system was real; the newest model stopped on its own. Anthropic says this is more a harness failure than an alignment failure, but the two companies they managed to reach didn’t notice anything until Anthropic called them. But there is another major reason that we should slow down on AGI. If model be
document fetched 2026-10-24 10:05