Periodic Labs: A chatbot walks into a laboratory

extract

“Most AI has been trained on the final artifacts of science, like the final paper, the final recount in a textbook. That isn’t often how the scientific process unfolded – it’s sort of a retelling of the story.” –Liam Fedus, Co-Founder of Periodic Labs Liam Fedus and Dogus Cubuk are the co-founders and co-CEOs of Periodic Labs, a startup building “synthesis superintelligence.” In practice, that means an AI system that closes the loop between hypothesis, experiment, and learning. Liam previously led post-training at OpenAI, where he was one of the creators of ChatGPT. Dogus spent years at Google DeepMind leading a large team of chemists and materials scientists. Together, they’ve brought those two trajectories to bear on a problem that has stymied science for decades: not just discovering new materials, but making them reliably and at scale. In this conversation, we explore the limits of language models, the commercialization bottleneck in materials science, why Periodic chose high-temperature superconductors as its first target, and what Liam learned from watching ChatGPT go from an internal product to one of the most successful products of all time. From my notebook We’re trying a new thing where I share a few of my thoughts from after the episodes. The moments that struck me, the tidbits that reframed my thinking, or something I just really enjoyed. Let me know if you like reading these! - To do science you have to do science. This is Periodic’s conviction in a nutshell. Science is the discovery of new things; you cannot prove it in a corpus of existing information. It has to involve trial and error in the real world. - Sci-fi remains an endless source of alpha. Why did Periodic set an initial goal to discover a new high-temperature superconductor? In part because ~every sci-fi future features it. (It also doesn’t hurt that it’s stayed top of mind in condensed matter physics.) - I underestimated the gap between discovery and utility. It can take 10-20 years for a new material to get to industrial scale. This feels like an equally important place for AI. We don’t want a century of discoveries stuck in twenty year manufacturing bottleneck. - Virality is ~impossible to predict. With the benefit of hindsight, it’s insane to hear Liam say that people inside OpenAI expected ChatGPT to attract tens of thousands of users. Perhaps the closer you are to something, the harder to know how useful it will be. - Capability ≠ virality. Another interesting wrinkle on the ChatGPT story. Liam said that they had a stronger internal model that hadn’t gone viral internally. ChatGPT used a weaker one! - Towards fuller observability of science. When humans conduct experiments, they write up the parts they believe matter. The ‘useless’ data you generate at every other step is discarded. This feels like one of the coolest parts of Periodic’s approach. In theory you can capture the full experimental trace. - Little things are big things. A bunch of samples get mislabeled in a lab. Usually, that would take a bunch of irritating human time to unwind. At Periodic’s lab, AI found it and fixed it. - “Contamination-free” RL is sort of an experimental veil of ignorance? Liam talked through one of Periodic’s advantages, which is constructing “snapshots” of experimental states. They can then ask: “What should we have done next?” This is possible because they have a corpus of fresh, untainted data. As a result, the model hasn’t inherited the right answer. Listen on YouTube • Spotify • Apple Podcasts Tangents What I’m into this week: - Nuclear War: A Scenario by Annie Jacobsen. Reading for…oh, no reason. - DJI Mic Mini plus WisprFlow is a pretty perfect combination for dictation. - Instinct is the best AI proof of concept I’ve seen in ~6 months. Good reasons to tread lightly, but the utility is incredible —Mario Every timestamp (00:00) Intro (02:05) Periodic’s mission (03:14) What it takes to build a synthetic superintelligence (04:48) Why “thinkism” falls short (05:46) Why Periodic is pursuing high-temperature superconductors (07:21) How Heike Kamerlingh Onnes discovered superconductivity (09:36) Why synthesizing new materials is so difficult (16:44) Dogus’s path to AI (20:18) Liam’s path to AI (23:28) Lessons from launching ChatGPT (25:24) Why Dogus and Liam chose each other (27:01) How Periodic attracts top talent (29:50) Inside an experimental loop at Periodic (35:10) What AI learns across experiments (40:10) What Periodic builds versus buys (44:37) Balancing diversity and depth across experiments (49:23) The difference in LLM performance on math vs. science (51:26) Why Periodic doesn’t depend on better models (53:20) Periodic’s short-term goals (54:09) The competitive landscape (55:39) Final meditations Liam and Dogus’s complete reading list - The Beginning of Infinity: Explanations That Transform the World (recommended by Liam): https://www.amazon.com/Beginning-Infinity-Explanations-Transform-World/dp/0143121359 - Subtle Is the Lord: The Science and the Life of Albert Einstein (recommended by Dogus): https://www.amazon.com/Subtle-Lord-Science-Albert-Einstein/dp/0192806726 Other resources - Periodic Labs: https://periodic.com - Bell Labs: https://www.nokia.com/bell-labs - QBasic: https://en.wikipedia.org/wiki/QBasic - Google DeepMind: https://deepmind.google - Google Brain: https://en.wikipedia.org/wiki/Google_Brain - OpenAI: https://openai.com Where to find Liam Fedus LinkedIn: https://www.linkedin.com/in/liam-fedus-26547811 Where to find Dogus Cubuk LinkedIn: https://www.linkedin.com/in/ekin-dogus-cubuk-9148b8114 Subscribe to the show I’d love it if you’d subscribe and share the show. Your support makes all the difference as we try to bring more curious minds into the conversation. Production and marketing by penname.co. For inquiries about sponsoring the podcast, email [email protected]. Transcript Mario Gabriele: AI has gotten very good at writing software. That’s no accident. After all, code has an answer key — a computer can check it in a fraction of a second. The program either runs or it doesn’t. Physics has an answer key too. The physical world too, but consulting it means running actual experiments, something an AI model can’t do on its own. Liam Fedus and Dogus Cubuk are building a company to change that. They are the founders of Periodic Labs, a startup building infrastructure to create what they call synthetic superintelligence. Liam previously led post-training at OpenAI and was one of the creators of ChatGPT, while Dogus was a research scientist at Google DeepMind. They’ve brought their experience together to build what might be described as an AI scientist, capable of conducting thousands of experiments a day and learning from each one of them. In this episode, we discuss the limits of LLMs, why science is so hard, and what Liam learned from launching ChatGPT. I’m Mario, and this is The Generalist. Mario Gabriele: I’m really excited to have you both here — I’ve been really looking forward to this conversation. I think you’re building such a fascinating company, so maybe we just start there. Tell me a little bit about Periodic. What are you trying to do? Liam Fedus: Our goal is to build a system that can intentionally synthesize matter — we’re calling this synthesis superintelligence. Our core belief is that in order to do this, you need to make connection with the real world. A system like this isn’t going to emerge just from reading textbooks and papers. It’s really necessary to use simulations, model the physical world, but then actually carry out the experiments. Thinking alone is not enough. It’s necessary to get new information from the world, conjecture how it might be, get the experimental results back, and do this end-to-end loop. By training end-to-end against the physical world, new types of intelligence and AI systems will emerge. So rather than a system that’s just been trained on the body of final scientific literature, it’s trained on the scientific process itself. Rather than knowing the state of science, it knows how to do science. Mario Gabriele: I’ve heard one way you’ve maybe described the mission in the past is almost to build an AI scientist, and decomposing the different pieces of that puzzle through that lens. How do you think about, fundamentally, what a scientist does, and what you need to replicate? Dogus Cubuk: The traditional way to look at science is that you come up with a hypothesis, and based on the hypothesis you run experiments, and the experiments either validate that hypothesis or invalidate it, and then you update your hypothesis. What’s really critical for us here is the observation that science always tries to push the boundary of our understanding — which by definition means it can’t just rely on what we already know, because science is trying to figure out what we don’t yet know. That’s almost at odds with the traditional description of machine learning, where a model trains on a training set and does well on a test set drawn from the same distribution. There’s a clear tension between what we call machine learning and what we call science. That’s where we’re basically trying to make progress. How have humans gotten around this problem? Because at the end of the day, humans are also machine learning algorithms, and they’ve often figured out things they didn’t know before — they had to iterate, run experiments, write down a theory, get yelled at by reviewers, and then update. Einstein had to do this. Bardeen had to do this. So we feel like AI will have to do it too. We’ve been spending a lot of time and focus on building labs where an LLM and a simulation can iterate together. That’s our goal. Mario Gabriele: Liam, you mentioned this idea of “thinking” — for folks who maybe don’t know what that really means, can you tell us a little bit about it and parse the differentiation a bit? Liam Fedus: I think it’s this concept that an AI that just thinks for a very long time — if you extrapolate the inference tokens well beyond anything we’ve seen — can just do anything. One of our deep scientific goals is to discover new high-temperature superconductors, and we don’t think a system trained the way models are trained today will think for a billion or a trillion tokens and suddenly arrive at a new high-temp superconductor. We think that’s just not in line with how we’ve pushed knowledge and science forward historically. That’s kind of a core belief of ours: you need to connect it to the physical world and iterate. The data simply doesn’t exist otherwise. That’s how you actually build new knowledge, not just rehash existing knowledge. Mario Gabriele: High-temperature superconductors — how did you land on that as the right first step for this company? Dogus Cubuk: There are a couple of reasons — let me talk through a few of them. As Liam said, our goal is synthesis superintelligence, and if we can synthesize anything in the lab, we might as well synthesize the most impactful thing, which is a superconductor. That’s one reason. Another is: think of any of the sci-fi futures you want to imagine. They often involve superconductivity — quantum computing, fusion, lossless transmission of energy. These all have superconductors somewhere in there. And there’s a sense in which superconductivity is the most macroscale quantum-mechanical phenomenon there is. There’s something very fascinating about it. If you think about the history of condensed matter physics, it goes through these roughly seven-year cycles of what’s popular, and then it falls out of fashion — topological insulators, heavy fermion physics. It kind of matches the length of a PhD; these topics come and go. Superconductivity is one topic that’s never gone out of style. Since its discovery by Onnes back in 1910, people have been fascinated by it, and we’re still fascinated by it today. Liam Fedus: And in the pursuit of this, you have to build up such sophisticated simulation infrastructure and AI infrastructure — and that has a huge amount of generality. Mario Gabriele: So it gives you the right stair-steps, and it’s also massively impactful. Liam Fedus: Exactly. In the pursuit of this super-ambitious goal, you build up a huge amount of incredibly impactful technology. Mario Gabriele: Dogus, I’ve heard you mention Onnes before. For folks who don’t know that history — what he achieved — can you tell us a little bit about why he’s an inspiration? Dogus Cubuk: Yeah, absolutely — he’s an inspiration to us in many ways. In the 1800s, there was a big rush to liquefy as many gases as possible. Around 1850 or so, people liquefied oxygen, for example, and that ended up playing a big role in industry. They were trying to go down the periodic table and liquefy as many of the gases as possible. As you can imagine, you had to lower the temperature more and more. Helium was the hardest one, because it’s a noble gas — it doesn’t interact with other atoms very strongly, which means its liquefaction temperature is very low. And helium, especially among the noble gases, is the smallest one. So that was the last one to be tackled, and Onnes built a lab to go after it. One of his interesting ideas was that science should be done at an industrial scale, by professionals — he hired professional technicians and engineers to work in a science lab, which people at the time found a little strange. And he succeeded. After a lot of hard work, with a large team, he was able to liquefy helium. Interestingly, he got the Nobel Prize for liquefying helium — not for discovering superconductivity. He’d basically created a new capability: lowering the temperature to below four Kelvin. Zero Kelvin is absolute zero — you can’t go below that — and four Kelvin is really cold. Once he reached four Kelvin, he could push a bit further, to 1.8 Kelvin. After that, he knew people were wondering what would happen to metals if they got really cold. Some really smart people, like Lord Kelvin, thought that at such low temperatures, electrons would basically freeze — which is intuitive, in a way; it’s so cold, electrons don’t want to move anymore. So Onnes wanted to try it, and one of the scientists in his lab tried putting different metals into liquefied helium. One of the things they tried was mercury. It turns out mercury is a superconductor at such low temperatures. Once they observed that, it was a big deal — and that opened up this whole field around superconductivity. Mario Gabriele: Thank you for that history, that’s very interesting. You’ve both used this phrase “synthesis superintelligence.” I’m sure folks have an understanding of what synthesis is, but why that word? Why is that the right wrapper, the right description? I imagine there’s something quite deep there in why you’ve chosen to frame it that way. Dogus Cubuk: There are historical reasons and scientific reasons. One historical reason is that the US used to be really good at synthesis, especially during the Bell Labs era. After Bell Labs stopped doing research at that scale, the US kind of lost its strength in synthesis, and it moved to some other countries. Mario Gabriele: When you say Bell Labs was great at synthesis, what’s an example of that? Dogus Cubuk: Good question. Say there’s a new compound, a new material that’s interesting, and I come to you and say, “Mario, can you make this?” That can be very hard. If it’s been made before, you can just follow the previous recipe — kind of like baking. But if it’s never been done before, it’s actually hard to figure out, because at a high level you’re trying to get the atoms into the right shape. And that’s difficult, because you’re not actually seeing the atoms — you can’t move them individually. Today, US academia has focused a lot more on functional properties, like finding a new battery or a new solar cell. But there’s this fundamental capability that’s still very important: being able to synthesize things. Here’s another reason: in materials discovery, you might have heard of this roughly twenty-year bottleneck, where you discover a material in the lab, but it usually takes ten to twenty years before it impacts technology. One of the main reasons is that you can make it in the lab, but figuring out how to scale up manufacturing — so you can make it at industrial scale and sell it — is usually a huge issue. A lot of what you see in the lab doesn’t transfer to large scale. And to be honest, people often think of materials discovery as being mainly about discovering the material. But if you go into the lab, usually the hardest part is making the thing you’ve discovered — it’s relatively easy to figure out what could be an interesting material. What’s usually hard is making it. Let me give you a quick example. When Alex Müller wanted to study transition metal oxides for new superconductors, he succeeded, around 1985, and discovered cuprates, which were the first unconventional superconductors — he got a Nobel Prize for it. These were the high-temperature superconductors. He’d actually originally been trying nickelates instead of cuprates — if you look at the periodic table, nickel is right next to copper. So instead of copper oxide, he tried nickel oxide. It didn’t work, so he gave up and moved to cuprates. But then, forty years later, our neighbors at Stanford — Harold Hwang — discovered that nickelates are also superconductors, once he figured out how to make them. I guess what we’re trying to say is: even when discovering a material is the main goal, being able to synthesize it efficiently is the biggest bottleneck. So we’re spending a lot of our time figuring out how to make the things we want to make. Mario Gabriele: For someone who last conducted a scientific experiment probably in tenth grade, and nothing serious — when you say it’s hard to make things, what are the ways that’s difficult, precisely? Is it a question of precision? Consistency? Pick any example, I suppose — what are the most common ways that process fails? Liam Fedus: Even if you know how you want to configure the atoms — okay, if they were arranged this particular way, it would be stable, it would have the properties we care about — there’s a whole set of processes you need to go through to get there. What elemental precursors do you use, in what proportions, what temperature profile, what pressures? What are all the processing conditions? There’s a huge amount of challenge in all of that, and it’s necessary to actually produce a new material. It’s insufficient to just bring the precursors together at ambient pressure and temperature — you have to push into new regimes. That’s extremely difficult. But once you have a system that can do this, it can intentionally and efficiently start designing matter, which we think is extremely foundational. Mario Gabriele: Another naive question — once you have the recipe, so to speak, for one of these things, how easily can you make it consistently? It sounds like once you have the idea, it’s hard to get it right the first time — but once you’ve dialed that in, is it quite replicable? Or is it actually surprisingly hard? Dogus Cubuk: It really depends. Let me give you some examples of why it’s hard to replicate. Take furnaces. If you think of a very simple synthesis — solid-state synthesis — you mix precursor powders, put them in an oven or furnace for some time at some temperature, and take them out. That’s the simplest thing you can do, and even there, there are a lot of issues. One issue is that precursors might have different amounts of impurities. You buy a precursor that’s, say, 99.1% pure, but there are other impurities, and those impurities can affect the results. Another issue is that furnaces change their temperature stability over time. One thing we realized in our own lab — a real pain point — was that a furnace was working well for the first three runs, but after that its temperature stability went haywire. Things like that can make replicability difficult. But humans have definitely gotten better at this with experience. Another example, from cuprates: when they were first discovered to be superconductors, some people wondered if it was even real, because people had trouble replicating it in the lab — it was a difficult synthesis procedure. Today, people have dialed it in so much that it’s in textbooks — if you open a solid-state chemistry textbook, some of the synthesis examples will include cuprates like YBCO. So it’s one of those things that seems unreplicable and hard to figure out at first, but as we gain experience, we get better at it. Liam Fedus: And one key piece of what we do at Periodic is that we think about the hardware engineering side of all of this too. Something that used to be a hidden variable — like an oven degrading slightly over time, or asymmetries within the furnace — if you can get better telemetry on these things, you can pipe that information back to the system, and it can build a better sense of it. Mario Gabriele: One of the things I think about when we talk about the state of modern science and modern knowledge is the replication crisis — it’s hard to reproduce results we think we know, so there are things we don’t necessarily know are true, or don’t know with full confidence. It sounds like because of that telemetry, and the way you can engineer this system with a different level of fidelity, you can hopefully start to figure out why these things fail, and get better replication and reproducibility. Liam Fedus: Yeah, and we also run replicates. We have to understand the variance of different systems, and really teach the system to make better decisions under uncertainty. For example, if there’s some aberrant noise in one pattern, we don’t want it to over-index on that and say, “No, this should just be rerun — this looks like noise.” Building that kind of scientific intuition into these systems is what we want to do. Mario Gabriele: I want to go so much deeper into what you’ve built, but maybe let’s take a step back in time first. I think you’ve both said before that you met at the gym, flipping a tire. How did you both get to that point? Maybe we could start with you — you were telling me before we started that you grew up in western Turkey, I believe. Dogus Cubuk: Yeah, that’s right. Growing up, I always knew I wanted to be a physicist, but what I didn’t realize at the time — though it makes more sense looking back — is that I always enjoyed simulating physics and math. If I say “QBasic,” do you know what that means? Mario Gabriele: I will not. Dogus Cubuk: Okay, some of the listeners will. QBasic was this programming language available when I was a kid, and as soon as I got my hands on it and figured out how to program, I was doing simulations of applied math and physics — though at the time it didn’t quite click for me. Back then I was really just thinking about theoretical physics; my heroes were people like Einstein and Feynman. In high school I learned more languages, like Java, and again it was all simulation-related. When I started my PhD, I realized something: physicists were using tools from the sixties and seventies to analyze data, but the amount of data had increased exponentially, because of Moore’s Law — people in the nineties had exponentially more compute than people in the seventies. So physicists were producing enormous amounts of data but analyzing it by hand, or with simple analytic models. This was around 2010, and I felt like we should see how machine learning could do physics. There was a lot of pushback at the time, because machine learning is probabilistic, it’s biased, it’s not always calibrated — and physicists, especially the Platonist ones, tend to think there’s one reality, one beautiful theory. So there was pushback. But in 2012, when the ImageNet moment happened and deep learning started gaining prominence, by the time I finished my PhD, machine learning was everywhere. Physics departments actually had to put machine learning somewhere in a grant proposal — especially in condensed matter physics — just to get funding. It was a huge change in about five years. Around that time, Google Brain had this residency program — I think it was partly Ilya’s idea, which was brilliant — to bring in people with some machine learning background or interest, but expertise in other fields, to learn deep learning and contribute to it at Google Brain. That’s how I joined, as part of the residency program. Liam joined in a similar way — he can explain that himself. So we were both doing deep learning research, and I was at the gym trying to keep up with a workout. It was difficult, so I picked him as the strongest person I could find. Liam Fedus: Yeah — you saw the arms, right? “This guy’s going to help me.” Dogus Cubuk: And it succeeded. Liam Fedus: You know, we managed. Dogus Cubuk: So I kept doing regular deep learning research — this was an exciting time, I’d just published at NeurIPS, ICLR, ICML — deep learning was advancing through these conference publications, and there was a lot of fun in that. I was also still pursuing my earlier passion, simulating solid-state physics. At some point, Liam and our other friend Barret Zoph left to start ChatGPT, and I ended up staying and really focusing on physics. By the time I left, I was running a large team of chemists and materials scientists at Google DeepMind, and we were doing a lot of simulation and machine learning work. Liam and I stayed connected, often talking about physics and other things, and we felt like the technology had reached a place where we could really impact the market and the industry. So we left our respective roles to start the company. Mario Gabriele: Amazing. Okay — what’s your origin story? Liam Fedus: I grew up in Maine, and I also had an early love for physics. In college I was torn between physics, chemistry, and economics, and I was part of a dark matter detection program. A key piece of that was reconstructing particle tracks, which became a physics model — really a physics-informed machine learning model. So I started doing machine learning in physics at a very early stage, in undergrad, and then again in graduate school, reconstructing particles, this time at CERN. Through that work, we kept pushing the amount of computation we put into it, and the algorithms for machine learning. But after a while it became clear that if I really wanted to push machine learning forward, I should be doing it in industry. So I went to Google, on the Google Accelerated Science team, and later Google Brain — that’s where Dogus and I met. I worked on a mix of things in the science program, then switched to generative models and reinforcement learning, and was really in a scaling mentality. It wasn’t necessarily applied to physics, but I saw how impactful language models were becoming, and worked on some of the architecture needed to scale them up — sparsity, Switch Transformers, mixture-of-experts. That became an important architecture for serving today’s models. We continued scaling that up, did some of the first trillion-parameter models. At some point it became clear there was a huge possible impact for this in the broader world — in a product. That’s what led me and a few others to OpenAI. We felt it was a great time to start building products with some of these things. When we arrived, GPT-3.5 was out as an API, GPT-4 was internally available, and we had some time to develop it. At some point we decided to do a quick sprint using GPT-3.5 — a two-week sprint — and that turned into ChatGPT. Mario Gabriele: And did that product do well? Liam Fedus: It exceeded some of our expectations. We had internal polls about how many people would use it — people guessed numbers like 10,000, 50,000. A high bar was a million. No one guessed it would be what it became. It was genuinely a low-key research preview. Mario Gabriele: Your mental model must have then totally shifted, right? I mean, that happened to literally all of us, but you were — Liam Fedus: Well, the context before that too is that Meta had just released Galactica and had to withdraw it within a few days of release. Mario Gabriele: I didn’t remember that. Liam Fedus: That was only a couple of weeks before, and there had been no successful chatbot up to that point. So the prior that this was going to be valuable technology was quite low. It was further surprising because we had GPT-4 internally, and it wasn’t this hugely viral product within the company. So putting out a weaker model to the public was really just part of iterative deployment. And obviously it worked better than our predictions. Mario Gabriele: What was the period after release like for you? Was it just totally frenetic — chaos? Liam Fedus: Yeah, at times. The servers couldn’t keep up, and you’re just trying to rapidly improve it. The early model couldn’t do math, couldn’t write very well, had many deep flaws — very different from what we have today. But it was just a series of getting feedback, improving the task distribution, improving data quality — all these kind of boring details — that led to the system we have today. Mario Gabriele: Very few people will ever have an experience like that, I suspect, but I imagine it serves you well as a founder today. What do you think it taught you? What did it give you? Liam Fedus: Question everything. It’s really interesting — people would come in years after the initial creation, and it’s very easy to anchor on some random decision made a year or a year and a half earlier, not necessarily an optimal decision, but you take it as one of the assumptions you build on. But I can recall, no — that was just one conversation, one decision someone made — feel free to change these things. As we were building that up, and eventually I was running the full post-training team, which sits between research and product, we made incredible progress in the digital world. But Dogus and I were talking, and we felt th