loading
loading
“Before fire existed, it’s hard to imagine all the implications of fire. I think the same thing will very much be true with the level of robotics that will be possible. It’s just hard to even think of all the possibilities.” – Pete Florence, Co-Founder & CEO of Generalist Pete Florence and I began by agreeing that his company has an excellent name. Generalist builds AI for robots, including models that can begin learning a new task from a single demonstration. We talked about dexterity, the strange business of gathering physical experience for a machine, and what we might build if robots could do much more of the work. Listen or watch on YouTube, Spotify, or Apple Podcasts. From my notebook We’re trying a new thing where I share a few of my thoughts from after the episodes. The moments that struck me, the tidbits that reframed my thinking, or something I just really enjoyed. Let me know if you like reading these! - Dexterity, dexterity, dexterity. Before the conversation, I was trying to think about what robotics’ equivalent of coding would be. I found myself looking for a big industry or use case – something in manufacturing or construction. Pete’s answer is dexterity. Like coding, it’s a horizontal capability that runs across fields and opens up a huge range of work. It’s the kind of simple, sharp description that made me realize I’d been thinking too narrowly. - Robots that build robots. There’s an implication of dexterity that I hadn’t properly thought through, but is obvious in hindsight. Once robots are truly dexterous, they will become capable enough to replicate themselves and successive generations. Your ability to manufacture starts to scale with the machines themselves. - GPT-3 feels roughly right. This is the comparison Anthropic researcher Sholto Douglas made when Generalist’s latest model launched. That feels apt. ChatGPT ran on GPT-3.5, and OpenAI had already trained GPT-4 by then. What Generalist is doing is remarkable, but we’re still short of that astonishing moment when everything clicks. - What might we build? I liked Pete’s framing of robotics opening up entirely new human projects. There are plenty of dangerous and unpleasant jobs we machines will take take over. But if the cost of physical labor falls dramatically, what kinds of strange, miraculous buildings, cities, and infrastructure become possible? What would we attempt that we currently dismiss as absurd? - Sometimes the fastest way is a plane. The lengths Pete has gone to get robotic data are extraordinary. He described packing hard drives into a backpack and getting on a plane because poor internet connectivity at the collection site meant flying would be faster. The team has also built custom servers and software to upload data from large numbers of memory cards at once. I loved getting a more visceral sense of the physical work that goes into this part of the process. Pete’s bookshelf When I asked Pete to recommend a book, he turned to two coaches: Bill Walsh of the San Francisco 49ers and John Wooden of UCLA. Walsh’s book is The Score Takes Care of Itself, a classic Wooden tome is On Leadership. Pete’s answer brought us back to teaching and coaching, both of which have been big parts of his life. Tangents What I’m reading, watching, and into this week: - Training a small model to autocorrect my text messages more accurately. So far…the gains are modest :) - Empire of Pain by Patrick Radden Keefe. I don’t usually read non-fiction in my spare time, but I’ve “enjoyed” (a strange word for so bleak a story) this excellent reporting of the Sackler family and the opioid epidemic. - Playing with the models’ 3D capabilities to spin up an Italo Calvino-themed personal website (in progress). —Mario Every timestamp 00:00 Intro 02:24 GEN-1.5 and one-shot learning 08:55 What makes a task difficult 10:53 Dexterity in robotics 14:53 Matt Mason’s definition of a hand 17:34 Giving robots a sense of touch 20:30 Growing up in a family of technologists and scientists 22:46 What drew Pete into robotics 27:38 Generalist’s founding thesis and what sets it apart 32:17 Protecting simplicity 35:01 How Generalist’s data collection has evolved 35:30 Generalist’s scaling philosophy 39:53 Generalist’s build-versus-buy philosophy 42:15 Balancing general intelligence with commercial deployment 44:24 Why Pete is betting on dexterity 47:33 The durability of the robotics application layer 49:09 Open-weight robotics models and data flywheels 50:52 Geopolitics, safety, and alignment 52:42 The parallels between teaching humans and robots 1:00:05 The timeline for robotics progress and adoption 1:03:37 Book recommendations Listen or watch on YouTube • Spotify • Apple Podcasts Pete’s complete reading list - The Score Takes Care of Itself: My Philosophy of Leadership: https://www.amazon.com/Score-Takes-Care-Itself-Philosophy/dp/1591843472 - Wooden on Leadership: How to Create a Winning Organization: https://www.amazon.com/Wooden-Leadership-Create-Winning-Organization/dp/0071453393 Other resources - Generalist: https://generalistai.com - GEN-1.5: https://generalistai.com/blog/gen-1.5 - Proprioception: https://en.wikipedia.org/wiki/Proprioception - Cutaneous receptor: https://en.wikipedia.org/wiki/Cutaneous_receptor - Fukushima nuclear accident: https://en.wikipedia.org/wiki/Fukushima_nuclear_accident - The bitter lesson: https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf - GEN-1: https://generalistai.com/blog/gen-1 Where to find Pete Florence LinkedIn: https://www.linkedin.com/in/peteflorence Website: https://www.peteflorence.com Subscribe to the show I’d love it if you’d subscribe and share the show. Your support makes all the difference as we try to bring more curious minds into the conversation. Production and marketing by penname.co. For inquiries about sponsoring the podcast, email [email protected]. Transcript Pete Florence: 2011 Fukushima nuclear disaster. News clip: The Fukushima number one nuclear power plant has been hit by its biggest earthquakes. Simply heartbreaking. Pete Florence: There wasn’t a robot in the world that could walk into the nuclear facility after the meltdown and just turn a valve, and so we had to send people in for that. If you’re not touching and moving the world and you’re a robot, what are you doing? Any process in the world that is gated by physical progress, building houses, whatever it is, we could just do that way faster, way better. Gen 1.5, it has a couple different capabilities where you can learn a physical task in just a single demonstration. The success rates of all these are modest today, but the fact that these success rates are anything above zero is the incredible thing. Mario Gabriele: Sholto Douglas, well-known Anthropic researcher, called this sort of a GPT-3 moment for robotics. Pete Florence: The example that we highlighted is using a brush to sweep an object into a bowl, and then instead you take the brush away and you give it a dustpan. Mario Gabriele: One of the stories of these past few years has been finding sort of the killer use case in coding. When it comes to physical intelligence, do you have an intuition of what, like, that killer use case might be? Pete Florence: I think the best analogy would be dexterity. If we can really crack dexterity, then the robots can build the robots and the robots can start building next-generation robots. Mario Gabriele: Robots don’t learn the same way that people do. Teaching a robot a new task can take months of programming, data collection, and retraining. Generalist, an AI company for robotics, is compressing that process down to seconds. In one of its recent demonstrations, a robot was shown just one time how to take money out of a wallet. It then completed that task immediately without needing to be retrained. Today’s guest, Pete Florence, is Generalist’s co-founder and CEO. The company is building what it calls general intelligence for the physical world: AI models meant to work across different types of robots, from factory arms to humanoids. Since its founding a little over two years ago, Generalist has grown into a key player in the robotics race. In this episode, Pete and I discuss one-shot learning, why dexterity remains so hard, where physical AI follows the same scaling laws as digital AI, and which industries change first when robots learn new skills much faster. I’m Mario, and this is The Generalist. Well, I think the only place to start is that I think you have the best name possible. Pete Florence: Likewise, Mario. Yeah. Mario Gabriele: Very good taste. Well, I’m really looking forward to this conversation and getting into what you’re doing at Generalist. It feels like a particularly exciting time. Earlier this month, I think it was mid-August—you released Gen 1.5, which is your latest foundation model, and it seemed to attract a lot of excitement. Uh, you know, I saw Sholto Douglas, a well-known Anthropic researcher, call this sort of a GPT-3 moment for robotics. Maybe you can tell us a little bit about what Gen 1.5 is and, and why it matters. Pete Florence: Sure. Um, so Gen 1.5 is, is definitely in a few different ways, like a type of model that we had been kind of waiting for, for a while, at least in terms of glimpses of types of capabilities that both are just exciting in and of themselves, and then also point towards like what might be a future of, of what’s coming next. Right. So overall Gen 1.5, it has a couple of different capabilities in what we call one-shot or a few-shot learning. Um, and also some other things in what we think are really remarkable levels of zero-shot physical generalization. These like one-shot and few-shot, broad types of learning capabilities where you can learn a physical task in just a single demonstration. And mind you, just to be clear, like the success rates of all these are modest today, but the fact that the success rates are anything above zero, yes, is, is the incredible thing. And, you know, if I, if I dial back like a few years, one of my co-founders, Andy Zhang and myself, like we’ve, we’ve been talking for a while about this threshold of crossing into, um, a very generalized one-shot and few-shot learning capability in robotics. And of course, like the rise of large language models around 2020—yes, of course, GPT-3—like that was the kind of the main thing that or at least arguably the main thing that, that people were especially excited about with the model GPT-3 was this broad one-shot and few-shot learning capability. They coined the term in-context learning in that paper. It’s, it’s been an incredible thing to see. Mario Gabriele: And so one-shot and few-shot learning is sort of a necessary principle to have that sort of like generalizable intelligence in the same way that GPT-3 had, where you’re able to say, hey, you know, make me a poem in the style of, you know, Emily Dickinson. And after a few tries, if you give it an example, it can do that. You’re sort of seeing that similar style of behavior in the physical world. Is that sort of the parallel? Pete Florence: Yeah. So, you know, specifically one, one-shot is where you, you give a single example of an input and an output. So you could say, like, “Who wrote Romeo and Juliet?” Answer: “William Shakespeare.” And that’s a, that’s like a question-answer, an input and output. And then you give another input and you let the model do the output. So you give it a single example and then the model, then it knows what to do from that single example. That would be like one-shot. You can also do zero-shot, which is you just, like, say, you know, “Who wrote Romeo and Juliet?” and you don’t give any, you know, example of a full input-output. Mario Gabriele: Yes. Pete Florence: Um, and it was actually, if you trace back the history, it was actually GPT-2 started to show a little bit of these types of capabilities, but it was really GPT-3 that like brought broad one-shot, few-shot learning into the, like, “Wow.” Basically, a very wide variety of tasks—you could just give it a go. You could just ask the GPT-3 model for those that remember, just like trying to prompt it with a few different examples, and it had a reasonable shot at doing something intelligent. And that’s like the level that we feel like we’re now seeing where the success rates are not super high. Um, there are limitations on the types of tasks that we can just prompt the model to do, but a very wide variety of tasks, you can just try to prompt it and see what happens. And there’s a reasonable shot that something, something intelligent occurs. And again, this like the fact that this happens at all is the incredible thing to us. Mario Gabriele: And the, you know, the tasks are things like take these bills out of a wallet or, you know, sort these, these cups, which it’s amazing to watch these things play out. Did you expect to see this capability as part of this model? Like, where are things tracking versus how you were sort of imagining they might? Pete Florence: It was surprising to us that this model started to show this particular type of one-shot visual capability, um, in part. And it’s something that we talk about a lot in, in what we released, um, describing some of the, you know, development behind the model, but it was not trained to do like one-shot or few-shot examples. And that is actually the same way that, um, at least as far as we know, as it is described in the GPT-3 paper that happened as well, where you actually just train the model, in the case of GPT-3, on every single possible piece of text that OpenAI could find at the time. And it’s later been studied and given a name that it’s kind of, you know, the way that the data is just laid out. It has this property which is called burstiness, which is that when you see a single SAT question example, just, just to give an example here, like, um, it’s pretty likely that your next, the next thing that the model sees is another SAT question. So like when you start to load the context of the model with similar types of things, it—oh, it’s pretty likely that the data distribution it was trained on has a similar type of thing occurring after it. That’s like kind of one of the ways that we can kind of try to hypothesize how this capability showed up in the model GPT-3. And similarly, like we never trained Gen 1.5 with like an example of a task and do it again. But the model just, you know, the physical world also has burstiness where if people are like taking bills out of a wallet, it’s pretty likely they’ll take another bill out of a wallet, right? Like it’s literally just, just, um, guessing what is most, yeah, most likely to happen next or what it should do next. Yeah. Based on a little bit of prompting in its context, it’s been surprising for us to see this model start to have that. I would have thought it would have taken maybe a little bit more time. Um, but at the same time, I, I do think it was inevitable at some point, maybe through different means to start to have this capability. Mario Gabriele: Do you have a sense of why it’s more successful at some of these tasks than others? Like, it seemed like I think the wallet, you know, example is actually it’s one of its best tasks, right? Versus, you know, things that to a human look sort of equivalently difficult. Is that just a data availability question? Is it something else happening? Pete Florence: That’s—it’s a great question. And the honest answer is it’s, it’s really hard to trace exactly why. We can sort of build up some intuition from a lot of the empirical experimentation. You know, in the pre-training distribution, you know, a lot of these tasks, there’s no way that, like every single task that you can think of is actually in your pre-training data. Like, definitely. Um, there are tasks that we can do that there was no pre-training data for this. And one of the easiest ways that we know this is we can just 3D print objects that we’re pretty sure have never existed before. We can demonstrate a task with it, and they can do it right. But there are certain sort of underlying skills or behaviors that together comprise a task. And if you have more data supporting those tasks, and that’s maybe one that would have higher success rates. But it’s not only—it’s not only about how much data is there. It’s not very simple to describe what makes a task challenging in robotics, especially for a dexterity or dexterous manipulation, but I think there’s a few different kind of component features that add up to make things challenging. Like one is how precise from a motion precision standpoint does a task have to be? How precise from a force precision standpoint? So how hard do you have to push on something, or how not hard you have to push on something? How coordinated do all of your degrees of freedom have to be? So like a task that really requires a lot of bimanual, precise coordination between the two different hands. That’s very challenging. Um, those are a few. But there are other aspects as well where it’s actually a—it’s a very rich space that we can try to kind of make sense of, about what makes the task challenging in the physical world. Those all contribute here. Yeah. Mario Gabriele: It’s—robotics is so interesting. You know this, you know, a thousand times better than I do. But I find myself getting so excited about some of the stuff that you’re doing because it touches on like what we think of—touches on biomechanics, it touches on like philosophy. You get into these evolutionary questions like when I was starting to do some reading about, you know, the evolutionary development of our own hands, it seems like it’s extremely—it traced extremely closely with the development of our cerebral intelligence as well as our manual intelligence, like those two things just, you know, came together in a really interesting way. You’ve mentioned before that dexterity is something that’s sort of like this, I don’t know, holy grail or big topic in robotics for a long time. More specifically, why is that something that seems to be just so important to get right? Is it just about the precision? Is it about those different factors that you sort of just mentioned? Pete Florence: So one for, for sure. You know, it’s something that honestly, both like me personally and the company has drawn a lot of inspiration from is exactly what you mentioned, which is that if you study evolutionary biology, um, you know, of course there’s evidence to suggest these things, but we can never know for sure. But, but it does seem that—and there’s this, there’s this term for this, like area of the brain. My brother’s a neuroscientist. It’s the Broca’s area or something like that. Mario Gabriele: Which is language. Pete Florence: Maybe it was believed to. There’s, you know, all these different regions of the brain can be associated with many different things. But there is some evidence that I have read in published research which suggests that, like the development of dexterity, um, and the evolutionary adaptiveness of like having better and better toolmaking skills then helped create the evolutionary pressure to have the right type of scaffolding in the brain, or even could have been the thing that helped just grow the brain in general. Yeah, that’s me extrapolating a bit from what I’ve read from scientific research, but there’s just the general connection between humans becoming tool builders and the drive to master more and more dexterity, and some suggestion that, that might have helped open up the brain for other types of cognition, like, like language and all these other things as well. The reality is that, of course, in, in a, in AI, like all these connections are, are sometimes nominal at best, but are also like genuine sources of inspiration. Right. Like the way that we have created AI today. Um, you know, digital neurons are not the exact same, of course, as neurons in our brain. But, but there is there’s a there’s a definite inspiration that has led to, you know, genuine, a significant advancement. So anyway, I think that, that’s just fascinating. But then over in the case of more just like grounded in what has been the state of robotics like over the past several decades leading up into now, and why, why dexterity is such a thing that stands out in general, I would say, and just the state of the state has been that very generalized dexterity, aka not just being able to pick up something that you have a predefined model of, and you’re doing it over and over again. But being able to touch and move the world and manipulate it a.k.a. Like move it into any type of configuration, that, that is the goal for the robot. That has been an incredibly challenging thing. And just the, the empirical, uh, state of research has been that it has been the hardest thing to master across a very wide range of tasks, whereas other things like, you know, in an amazing way, it wasn’t the case a couple of decades ago. Like walking robots work pretty well, at least on like pretty flat ground. And there are edge cases that, you know, general cases of what is called loco-manipulation or parkour type things are challenging, but walking around on flat ground has become a pretty well-developed capability. Meanwhile, putting a robot in front of a bunch of different objects and getting it to do anything with them, especially for challenging different types of objects, challenging different types of tasks. That has been—that has been a very elusive thing. Mario Gabriele: I always like to ask a few people close to the interview subject what I should ask you, and one thing that someone suggested was, “You should ask him. You should ask Pete how he would describe a hand. What does he think a hand is, uh, fundamentally?” Pete Florence: There’s this guy, Matt Mason, who’s, um, arguably like the godfather of robotics manipulation. He has a nice definition, which I’ll butcher a little bit. I haven’t read this in years, but it’s somewhat in my head, which is that robotic manipulation is the process, or something like that, of selectively choosing contact with the world to put the world into some desired configuration. And the operative part here is selectively making contact. Right. And it is actually very carefully chosen because you can do things like not touch something. And that can be your—that could be part of your goal. It’s not just making contact, but it’s also choosing to make contact or not. And there’s this very particular word that just for those that are maybe not acquainted with robotics—like, contact, you know, that’s a word that in many different contexts has many different meanings. Yes, but contact is really everything in dexterity or what’s called manipulation in research. It’s about touching something and being able to, you know, have things that are not touching, put them so that they are now touching. And when they are now touching, you can move them around. And it just turns out that when you have to actually touch and move the world, that is where the physics is really hard to predict. Um, it’s an incredibly rich sort of, at its core, a mathematical problem that, uh, you know, for many decades we’ve tried to, like, solve these types of things analytically. It’s very challenging. And in practice, the thing that has made the most progress, just like in language—and, you know, at this point, it’s, it’s, it’s just such a well-known story—but just having models that are trained on a massive variety of data and, very carefully, the training recipes are, you know, refined through tons of experimentation. That’s been the thing that has given us the most progress. So, sorry, back to hand. Did you ask me about this? Mario Gabriele: This is exactly the sort of rabbit hole I wanted to go down, so that’s great. Pete Florence: I think a hand you could say is anything that helps you selectively make contact or not is what I would say is the definition. So, you know, that’s pretty broad. Mario Gabriele: Yeah. Yeah. You can imagine the evolutionary pressures of our—of the hands of the future could be—could look very different. Pete Florence: Yeah. And even, like, you know, things like feet, we can consider hands. Yeah, yeah. And of course we can use our feet as hands. Mario Gabriele: How do you think about giving robots a real sense of touch? Like, is that pressure? Is it temperature? Like, what are the right ways to gather that kind of data? And what does it miss at the moment versus real human touch? Pete Florence: Human touch is incredible. Um, I haven’t studied human touch, um, that much, but of course, I have an intuitive sense of human touch. Um, and I think in robotics, touch is certainly important. A simple exercise on how much the like touch is limiting us is you can teleoperate a robot and you can have no touch feedback, and you’ll realize that, like, you pretty quickly could teleop a robot with no touch feedback at a very like super-AI level, not like superhuman level. Because you are, you can do it much better than, than our best AI models can today. Mario Gabriele: Yes. Pete Florence: And what that shows you is that like, yes, there are certain cases where touch would, would really help us a lot more. But at the same time, human brains still are quite incredible compared to even our best models, even Gen 1.5, etc., at generalized dexterous manipulation. That said, I strongly believe touch is important and will be part of like reaching the highest pinnacles of performance that we can with robots. Then there’s actually quite a lot of different options for how robots can effectively get these types of like non-visual or non-audio types of sensation of the world. I said before, I haven’t studied human touch that much, but I know a little bit like so one is that like there’s this concept called proprioception, which is um, sometimes people would say different than touch, but it’s very relevant here in that it’s like non-visual non-audio senses of the world, and we as humans have a bit of proprioception in that. Like we know how our arms are bent. We can close our eyes and we know how our arms are bent. Yes. And that one is, is pretty easy for robots to get. Basically every robot gets—gets that for free. Mario Gabriele: Yeah, that’s called encoders. Pete Florence: You know, you know your joint angles. Another type of proprioception is knowing how hard something is, like pushing on the muscle at your joint. And that one a lot of robots now also have either through what’s called current sensing or torque sensing at the joints, but that one is also like in most robots already today. And those two alone, I think, provide a lot of what you might think of as touch sensing. But then there are other things where I believe the term is called a cutaneous touch, or like the ability to like, you have your fingertip and you can like scratch it and you’re not feeling it like down in the rest of your arm, but you can feel it right there. That is the one where there’s starting to be more and more sensors there. I don’t think, um, it’s not at least a widely deployed thing to have lots of those sensors all over robots today. I think over time we might see more of those. And yeah, I think that there’s a lot of cases where those types of sensors will be useful. Mario Gabriele: You grew up in the Bay Area. Uh, were your parents in tech? Did you have models of entrepreneurship sort of around you all the time, or was that not sort of the version of your, your upbringing? Pete Florence: Both my parents worked in, uh, technology at different points. My mom’s side of the family has been in the Bay Area for a very long time. So I think this is actually like, um, you know, living in San Francisco, a lot of people come and go. I love San Francisco, but the reality is, a lot of people that you meet in tech, you know, uh, maybe moved here, uh, this generation or last generation. For my mom’s side of the family, it traces back to, like, I’m a sixth-generation Bay Area. Mario Gabriele: Oh, wow. Pete Florence: And it was my, uh, triple-great-grandfather, moved from Lafayette, Indiana, settled in the Bay Area up in Petaluma. That’s my mom’s side of the family. My dad’s side of the family—uh, my dad moved up from LA. Um, but, yeah, my, my mom worked in, uh, in tech, uh, and then my, my dad worked with a bunch of different startups, and my grandfather was the—was the scientist. He was a chemist at, you know, back in the middle of the 20th century, one, uh, frontier area of technology was, uh, rubber. And, uh, yeah, he was, he was a chemist, and he worked at a—at a rubber factory. Mario Gabriele: And was he sort of—I know you studied chemistry. Was he a bit of an inspiration in picking your sort of first paths? Pete Florence: Uh, I did study chemistry. That’s what I did in, in college. 100%. Yeah. I think my grandfather helped inspire that. Mario Gabriele: Were you the sort of kid that was, you know, programming, uh, you know, in your early days or, you know, building, building things in that sense or did that come, you know, a little bit later? Pete Florence: There was always building stuff. Uh, I was actually just talking with my wife about this the other day that, like, she was talking to my brother, and we were just talking about, like, Pete as a 12- or 13-year-old and, uh, modifying Nerf guns to make them like, uh, you know, more powerful. Uh, we actually me and my one of my best friends, uh, we had a, a website we called a modification directory for modifying Nerf guns. Mario Gabriele: Oh, I love it. Pete Florence: The same thing with Super Soakers. Those were like the types of things that I would, would, would do at that age. Um, and then yeah, programming. And in college I did, you know, enough to sort of like support, like, uh, scientific research that I was doing in chemistry or in physics. Um, but then graduate school was really when I, like, became a more full-blooded so
“Before fire existed, it’s hard to imagine all the implications of fire. I think the same thing will very much be true with the level of robotics that will be possible. It’s just hard to even think of all the possibilities.” – Pete Florence, Co-Founder & CEO of Generalist Pete Florence and I began by agreeing that his company has an excellent name. Generalist builds AI for robots, including models that can begin learning a new task from a single demonstration. We talked about dexterity, the strange business of gathering physical experience for a machine, and what we might build if robots could do much more of the work. Listen or watch on YouTube, Spotify, or Apple Podcasts. From my notebook We’re trying a new thing where I share a few of my thoughts from after the episodes. The moments that struck me, the tidbits that reframed my thinking, or something I just really enjoyed. Let me know if you like reading these! - Dexterity, dexterity, dexterity. Before the conversation, I was trying to think about what robotics’ equivalent of coding would be. I found myself looking for a big industry or use case – something in manufacturing or construction. Pete’s answer is dexterity. Like coding, it’s a horizontal capability that runs across fields and opens up a huge range of work. It’s the kind of simple, sharp description that made me realize I’d been thinking too narrowly. - Robots that build robots. There’s an implication of dexterity that I hadn’t properly thought through, but is obvious in hindsight. Once robots are truly dexterous, they will become capable enough to replicate themselves and successive generations. Your ability to manufacture starts to scale with the machines themselves. - GPT-3 feels roughly right. This is the comparison Anthropic researcher Sholto Douglas made when Generalist’s latest model launched. That feels apt. ChatGPT ran on GPT-3.5, and OpenAI had already trained GPT-4 by then. What Generalist is doing is remarkable, but we’re still short of that astonishing moment when everything clicks. - What might we build? I liked Pete’s framing of robotics opening up entirely new human projects. There are plenty of dangerous and unpleasant jobs we machines will take take over. But if the cost of physical labor falls dramatically, what kinds of strange, miraculous buildings, cities, and infrastructure become possible? What would we attempt that we currently dismiss as absurd? - Sometimes the fastest way is a plane. The lengths Pete has gone to get robotic data are extraordinary. He described packing hard drives into a backpack and getting on a plane because poor internet connectivity at the collection site meant flying would be faster. The team has also built custom servers and software to upload data from large numbers of memory cards at once. I loved getting a more visceral sense of the physical work that goes into this part of the process. Pete’s bookshelf When I asked Pete to recommend a book, he turned to two coaches: Bill Walsh of the San Francisco 49ers and John Wooden of UCLA. Walsh’s book is The Score Takes Care of Itself, a classic Wooden tome is On Leadership. Pete’s answer brought us back to teaching and coaching, both of which have been big parts of his life. Tangents What I’m reading, watching, and into this week: - Training a small model to autocorrect my text messages more accurately. So far…the gains are modest :) - Empire of Pain by Patrick Radden Keefe. I don’t usually read non-fiction in my spare time, but I’ve “enjoyed” (a strange word for so bleak a story) this excellent reporting of the Sackler family and the opioid epidemic. - Playing with the models’ 3D capabilities to spin up an Italo Calvino-themed personal website (in progress). —Mario Every timestamp 00:00 Intro 02:24 GEN-1.5 and one-shot learning 08:55 What makes a task difficult 10:53 Dexterity in robotics 14:53 Matt Mason’s definition of a hand 17:34 Giving robots a sense of touch 20:30 Growing up in a family of technologists and scientists 22:46 What drew Pete into robotics 27:38 Generalist’s founding thesis and what sets it apart 32:17 Protecting simplicity 35:01 How Generalist’s data collection has evolved 35:30 Generalist’s scaling philosophy 39:53 Generalist’s build-versus-buy philosophy 42:15 Balancing general intelligence with commercial deployment 44:24 Why Pete is betting on dexterity 47:33 The durability of the robotics application layer 49:09 Open-weight robotics models and data flywheels 50:52 Geopolitics, safety, and alignment 52:42 The parallels between teaching humans and robots 1:00:05 The timeline for robotics progress and adoption 1:03:37 Book recommendations Listen or watch on YouTube • Spotify • Apple Podcasts Pete’s complete reading list - The Score Takes Care of Itself: My Philosophy of Leadership: https://www.amazon.com/Score-Takes-Care-Itself-Philosophy/dp/1591843472 - Wooden on Leadership: How to Create a Winning Organization: https://www.amazon.com/Wooden-Leadership-Create-Winning-Organization/dp/0071453393 Other resources - Generalist: https://generalistai.com - GEN-1.5: https://generalistai.com/blog/gen-1.5 - Proprioception: https://en.wikipedia.org/wiki/Proprioception - Cutaneous receptor: https://en.wikipedia.org/wiki/Cutaneous_receptor - Fukushima nuclear accident: https://en.wikipedia.org/wiki/Fukushima_nuclear_accident - The bitter lesson: https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf - GEN-1: https://generalistai.com/blog/gen-1 Where to find Pete Florence LinkedIn: https://www.linkedin.com/in/peteflorence Website: https://www.peteflorence.com Subscribe to the show I’d love it if you’d subscribe and share the show. Your support makes all the difference as we try to bring more curious minds into the conversation. Production and marketing by penname.co. For inquiries about sponsoring the podcast, email [email protected]. Transcript Pete Florence: 2011 Fukushima nuclear disaster. News clip: The Fukushima number one nuclear power plant has been hit by its biggest earthquakes. Simply heartbreaking. Pete Florence: There wasn’t a robot in the world that could walk into the nuclear facility after the meltdown and just turn a valve, and so we had to send people in for that. If you’re not touching and moving the world and you’re a robot, what are you doing? Any process in the world that is gated by physical progress, building houses, whatever it is, we could just do that way faster, way better. Gen 1.5, it has a couple different capabilities where you can learn a physical task in just a single demonstration. The success rates of all these are modest today, but the fact that these success rates are anything above zero is the incredible thing. Mario Gabriele: Sholto Douglas, well-known Anthropic researcher, called this sort of a GPT-3 moment for robotics. Pete Florence: The example that we highlighted is using a brush to sweep an object into a bowl, and then instead you take the brush away and you give it a dustpan. Mario Gabriele: One of the stories of these past few years has been finding sort of the killer use case in coding. When it comes to physical intelligence, do you have an intuition of what, like, that killer use case might be? Pete Florence: I think the best analogy would be dexterity. If we can really crack dexterity, then the robots can build the robots and the robots can start building next-generation robots. Mario Gabriele: Robots don’t learn the same way that people do. Teaching a robot a new task can take months of programming, data collection, and retraining. Generalist, an AI company for robotics, is compressing that process down to seconds. In one of its recent demonstrations, a robot was shown just one time how to take money out of a wallet. It then completed that task immediately without needing to be retrained. Today’s guest, Pete Florence, is Generalist’s co-founder and CEO. The company is building what it calls general intelligence for the physical world: AI models meant to work across different types of robots, from factory arms to humanoids. Since its founding a little over two years ago, Generalist has grown into a key player in the robotics race. In this episode, Pete and I discuss one-shot learning, why dexterity remains so hard, where physical AI follows the same scaling laws as digital AI, and which industries change first when robots learn new skills much faster. I’m Mario, and this is The Generalist. Well, I think the only place to start is that I think you have the best name possible. Pete Florence: Likewise, Mario. Yeah. Mario Gabriele: Very good taste. Well, I’m really looking forward to this conversation and getting into what you’re doing at Generalist. It feels like a particularly exciting time. Earlier this month, I think it was mid-August—you released Gen 1.5, which is your latest foundation model, and it seemed to attract a lot of excitement. Uh, you know, I saw Sholto Douglas, a well-known Anthropic researcher, call this sort of a GPT-3 moment for robotics. Maybe you can tell us a little bit about what Gen 1.5 is and, and why it matters. Pete Florence: Sure. Um, so Gen 1.5 is, is definitely in a few different ways, like a type of model that we had been kind of waiting for, for a while, at least in terms of glimpses of types of capabilities that both are just exciting in and of themselves, and then also point towards like what might be a future of, of what’s coming next. Right. So overall Gen 1.5, it has a couple of different capabilities in what we call one-shot or a few-shot learning. Um, and also some other things in what we think are really remarkable levels of zero-shot physical generalization. These like one-shot and few-shot, broad types of learning capabilities where you can learn a physical task in just a single demonstration. And mind you, just to be clear, like the success rates of all these are modest today, but the fact that the success rates are anything above zero, yes, is, is the incredible thing. And, you know, if I, if I dial back like a few years, one of my co-founders, Andy Zhang and myself, like we’ve, we’ve been talking for a while about this threshold of crossing into, um, a very generalized one-shot and few-shot learning capability in robotics. And of course, like the rise of large language models around 2020—yes, of course, GPT-3—like that was the kind of the main thing that or at least arguably the main thing that, that people were especially excited about with the model GPT-3 was this broad one-shot and few-shot learning capability. They coined the term in-context learning in that paper. It’s, it’s been an incredible thing to see. Mario Gabriele: And so one-shot and few-shot learning is sort of a necessary principle to have that sort of like generalizable intelligence in the same way that GPT-3 had, where you’re able to say, hey, you know, make me a poem in the style of, you know, Emily Dickinson. And after a few tries, if you give it an example, it can do that. You’re sort of seeing that similar style of behavior in the physical world. Is that sort of the parallel? Pete Florence: Yeah. So, you know, specifically one, one-shot is where you, you give a single example of an input and an output. So you could say, like, “Who wrote Romeo and Juliet?” Answer: “William Shakespeare.” And that’s a, that’s like a question-answer, an input and output. And then you give another input and you let the model do the output. So you give it a single example and then the model, then it knows what to do from that single example. That would be like one-shot. You can also do zero-shot, which is you just, like, say, you know, “Who wrote Romeo and Juliet?” and you don’t give any, you know, example of a full input-output. Mario Gabriele: Yes. Pete Florence: Um, and it was actually, if you trace back the history, it was actually GPT-2 started to show a little bit of these types of capabilities, but it was really GPT-3 that like brought broad one-shot, few-shot learning into the, like, “Wow.” Basicall
document fetched 2026-10-24 11:25 · canonical https://generalist.com/p/pete-florence-when-robots-improvise