Runway: Learning From Watching the World

extract

Welcome to the 1,825 newly Not Boring people who have joined us since our last essay! Join 281,011 smart, curious folks by subscribing here: Hi friends 👋, Happy Tuesday! A couple of weeks ago, I mentioned that I was partnering with Runway this quarter as our presenting sponsor to bring some of my writing to life in the form of video. It took us a while to land on the right partner to form a deeper relationship with — it has to be a company I want to talk about a bunch — but when Runway reached out, it was a clear perfect fit. For one thing, I’ve had a lot of fun playing around with their video models. But for two others: 1. I’m fascinated by World Models (and quoted Runway co-CEO Anastasis Germanidis in the essay Pim DeWitte and I wrote) and 2. Runway has been on a tear with new product releases. I wanted to understand how, so to kick off the partnership, I asked to write an essay with me explaining how video, interfaces, operating systems, and robotics all fit together, why video was the right starting point for World Models, where the world is heading if Runway’s bets are right, and whether or not, as someone who simulates worlds, he believes we’re living in a simulation. Let’s get to it. Runway: Learning From Watching the World A Co-Written Essay with Runway Co-CEO Anastasis Germanidis From time to time, a certain company catches fire. You can tell when it’s happening by the pace and breadth of their product releases. One day, a new capability. Cool. The next day, another. Woah. Next day, another still. Runway, the world model company best known for releasing the first commercial video generation model, is on such a run. Over the past six weeks, it has shipped a real-time interface model, a new interactive world model, an open-weight robotics model, an AI agent for performance marketers, a research preview of an operating system, and it bought a robotics lab in Paris along the way. Meanwhile, the business is growing faster than at any point in its history, adding more than $100 million in annual recurring revenue in the last quarter alone. From the outside, it looks like a company that suddenly figured something out. From the inside, the view is much more interesting. Runway has been pursuing a particular set of bets for eight years, and that near-decade of research and iterative deployment is now yielding a seemingly disparate but actually related set of new models and products. On September 1st, Runway introduced Solaris, the “first Interface World Model,” which allows users to generate interfaces in real-time, through prompts or actions on screen, like this one where a guy takes a selfie and tries on glasses and tweaks his hair and smile with the drag of a mouse. On September 8th, Runway announced it had acquired Kinetix, a Paris-based research lab, which specializes in… robotics? From the video company? Then at their September 30th AI Summit, the team shared Praxis-1, “an open-weight World Action Model that turns our video pre-training expertise into real world control for robots.” The thesis is that while robotics data is limited, video data is practically infinite. So Runway pre-trains on video data and then uses that as a policy model for robotics developers who can train it on their specific embodiments. Standard Bots, whose co-founder Evan Beard and I co-wrote an essay on robotics, is working with Runway to deploy Praxis-1 on its arms. And then the very next day, on October 1st, Runway previewed Project Continuum, “a new operating system research application built around real-time video interfaces.” A video model. An interface model. A robotics model. An operating system. Are you getting it? These are not separate pretrains. They’re different applications, built around the same General World Model backbone, with different post-training on top. Made with Runway All of them are examples of what you can do when you can simulate any pixel. An operating system ends up as pixels on a screen, like video and interfaces do, and like the world does through a robot’s eyes. If you can simulate every pixel at high enough fidelity and speed, then you can make almost anything that shows up to the world as pixels. This is the future Runway is building towards. Runway is making a few different bets, on a research direction, the Bitter Lesson, and a business strategy. It is betting that world models can learn, like humans do, from third-person observation. It is betting that letting models watch a ton of video will lead to models that understand how the world works better than they could by reading about how the world works or from any other human-mediated data source. And it is betting that it can build a massive business in content creation in order to both help fund the large investment into better and better models, and to become the rare company that is both a frontier lab and a distribution layer. Runway’s commercial success and product prolificness points to a surprisingly wide set of applications for World Models; surprising, that is, unless you read the essay Pim DeWitte and I co-wrote explaining why World Models have the potential to eat everything. In it, we wrote: World models — these systems that learn from watching the world and the actions taken in it — are a fundamentally new kind of foundation model. They can compute what was previously uncomputable. There are a number of different types of models whose makers call them World Models, although many don’t fit our definition; they don’t predict the next state based on intervention. We categorized Fei-Fei Li’s World Labs, which AMD recently agreed to acquire for $8.2 billion, as building “3D Reconstruction and Generation Models.” Runway, we wrote: Is blurring the lines between video generation and world generation, as mentioned in the Video Model section. During its Research Demo Day 2025, Runway co-founder and CTO Anastasis Germanidis explained the company’s evolution that started from “generative AI models [as] viable tools for creative expression”. They then evolved towards World Models (while still making incredible progress in video models.) “To build a World Model,” Germanidis explained, “We first needed to build a really great video model. We believe that’s the right path to building World Models, that teaching models to predict pixels directly is the best way to achieve general purpose simulation.” Runway first mentioned its World Model ambitions in a December 2023 blog post by Germanidis, who is now co-CEO, titled “Introducing General World Models.” This has been the plan for quite some time: teach the models to make videos from watching videos, and then teach them how to bring the actions in video to life. Anastasis describes it as a plan in four chapters: Chapter 1: Media and Entertainment Chapter 2: Broader Content Needs and Full Video Chapter 3: Robotics (We Are Here) Chapter 4: Science The plan is coming together, as evidenced by the recent string of releases, and I have become a World Models convert who believes they have the potential to do a lot of things in the real world that LLMs can’t, so Anastasis and I decided to co-write an essay on the company’s bet that watching a lot of videos can teach models to do almost anything. Learning from Watching One of the things you realize when you have kids is that babies are learning machines. When they’re really young, they just keep changing, picking up new skills, like little Neos. The thing about babies, though, is that they don’t know how to read. They don’t learn like we learn in school, by reading textbooks, because they can’t. Instead, they watch. They watch how we walk, how we talk, how we smile, how we bring food to our mouths, how we pick up toys and how we put them away, they watch how we shake hands, and hug, and run, and jump, and crawl. Then they try it themselves, clumsily at first, they watch again, they try again, they watch again, they try again, and try and try and try until they’ve gotten it. Made with Runway As we grow up, we do go to school, and we do learn from books, but we still learn by watching. We watch how the kids in the older grades dress and talk and interact. We watch how our favorite athletes play. We watch to learn how to interact with and manipulate the world. If you’re a guy, think about the last time you had to tie on a bowtie for a wedding. The best way to do it, other than having the one guy who knows how to tie a bowtie tie it for you, is to watch that guy tying his bowtie. The second best is finding a YouTube video of someone walking you through how to tie a bowtie. At the bottom of the list is reading how to tie a bowtie. I’ve tried it, and it simply doesn’t work, at least not for me. Language is compression. We look at the rich, ever-changing world in front of us, and we can’t possibly capture it all. By the time we’ve come anywhere even close to describing what we see at any given time with any amount of precision, it’s changed. And no matter how hard you try, words will never be enough, because you can’t, for example, see each of the atoms in the scene, track the movement of the electrons and protons and neutrons that make up those atoms, see how electromagnetic and gravity waves interact with them… you get it. Everything in the world changes, then changes again, and each time it does we’d have to start describing it again, like verbal Sisyphuses. Instead of running a constant monologue as we walk down the street, describing all of the motion and appearances and interactions we see, we simply say, “The man walked down the street.” Neither approach captures the full picture, but the latter is easier and allows us to actually live our lives. To jack as many subtle details into our brain as possible, therefore, we don’t read about life, we watch it. Our brain has evolved to pull out the relevant information. Made with Runway A picture is worth a thousand words. A video is worth 1,000 times the number of frames plus however many words the motion connecting them and sounds are worth. But the value of video isn’t simply that it contains more information than words. Predicting what happens next and checking how well you predict, over and over again, creates a pressure to learn reusable relationships. Imagine trying to predict the next frame of that street video. To get it right, it isn’t enough to know what the man looks like. You need to know that he’ll keep walking in roughly the direction he was walking, at roughly the speed he was walking, that his left foot comes after his right, that people walking near him will walk behind him and not through him, that when he passes in front of a moving car, that car still exists and will come out the other side a beat later as the same car, that his shadow moves when he moves, etc. When a model is trained on video, nobody tells the model any of this. There’s no line of code that says “objects still exist when something walks in front of them” or “people can’t walk through people.” The model guesses the next frame, sees how wrong it was, and adjusts. Somewhere along the way, the cheapest way for the model to stop being wrong is to learn something like the rules: things fall, and seem to fall in basically the same way at basically the same rate of acceleration (although feathers seem to flutter back and forth while bricks simply drop); glass shatters and rubber bounces; a ball that rolls behind something comes out the other side looking exactly the same, unless there was a puddle of mud. One video with a muddy ball might confuse the model, but over millions of them, it would learn that the mud is an anomaly. These, incidentally, are things that babies learn during their first year. We play peek-a-boo because it makes them laugh, but also because it teaches them object permanence, this idea that things still exist even when you can’t see them. This is how LLMs work, too. Their goal is to predict the next word, by any means necessary. Ilya Sutskever has been making this point for years: predicting the next token correctly requires understanding something about the reality that produced the token. Swap tokens for frames and you have Runway’s thesis. There’s a theory in neuroscience, which I discussed with Cristóbal Valenzuela, Runway’s co-founder and Anastasis’s co-CEO, called active inference. Karl Friston is its most well-known proponent. The idea is that your brain is a prediction machine, constantly running a simulation of what’s about to happen and checking it against what actually happens. You don’t normally notice it, because you get really good at prediction and it just runs in the background. You only notice it when you get it wrong. You’re walking up the stairs, you go for one more step, the step isn’t there, and for a split second your whole body lurches while the simulation crashes and reboots. “The moment you see it didn’t predict,” Cristóbal said, “you adjust.” A model that sees enough examples of success and failure starts to carry around its own version of that simulation. But until the model runs in a physical body instead of a screen, it can’t feel what it’s like to stumble, trip, or fall. Why video, though, and not another type of data? That’s where the first of Runway’s bets comes in: The Bitter Lesson. The Bitter Lesson and the Big Blob of Video It is practically illegal to write about training models without mentioning The Bitter Lesson. In 2019, Richard Sutton wrote a short essay arguing that the biggest lesson from seventy years of AI research is that “general methods that leverage computation are ultimately the most effective, and by a large margin.” It is bitter because every time researchers have tried to build their own understanding of a domain into a system (chess openings, the structure of grammar, the edges and corners in an image), they’ve eventually been steamrolled by dumber methods that could soak up more compute and more data. For his part, Anthropic CEO Dario Amodei made a similar observation two years earlier, in a recently-partially-leaked internal OpenAI research document he titled “Big Blob of Compute.” The Big Blob of Compute (BBOC) hypothesis, he wrote, “says that creating any given intelligent behavior is mostly about providing a large, minimally structured mass of computational capacity, and then giving it shape and form via interactions with a rich environment and a training process that drives towards behavior appropriate for the task at hand.” Runway’s corollary is that the most scalable source of data tends to win. Call it the Big Blob of Video (BBOV) hypothesis. LLMs won on text because the internet had an absurd amount of it. Robotics has been climbing its own ladder. It used teleoperation data, a specific robot doing a specific task puppeteered by a human. This year, many robotics labs began using a lot more egocentric video from cameras strapped to people’s heads as they perform a task. Runway is betting on something simpler and less human-mediated: vast quantities of video that already exist, and will be filmed (and generated) from here on out. Watching video is the most Bitter Lesson-pilled approach to AI there is. Language is a compression of reality, an incredibly good one that humans have been refining for thousands of years, but a compression nonetheless. As Sutton wrote, “We want AI agents that can discover like we can, not which contain what we have discovered. Building in our discoveries only makes it harder to see how the discovering process can be done.” In other words, we limit agents with our own beliefs. Put more bluntly: if you really believe the Bitter Lesson, you should be working on World Models. There’s a good argument against this thesis, so we might as well make it ourselves. If language is compression, well, compression can be useful! Learning about the world from pixels is less efficient than learning from text. Researchers have shown that you can build a language model out of raw audio, just by predicting the next little chunk of sound, and that language eventually emerges, but that it takes something like three orders of magnitude more compute than learning from text. Text comes predigested. Somebody already did the work of sorting out the most important stuff. Pixels make the model do that work itself, and there are a lot of pixels on which the model must do it. Then there’s Yann LeCun’s critique: predicting every pixel wastes effort on details that don’t matter (every leaf on every tree, every ripple in every puddle), so models should predict in an abstract representation space instead. Runway’s answer is pragmatic: use whatever works. So far, we haven’t seen evidence that JEPA scales better than the diffusion models it uses, and the simplest approach tends to scale best. Plus, it’s really hard to know ahead of time which pixels are noise. The rustling leaf isn’t noise if it tells you that the wind has picked up. The wind, even ever so slightly, will impact how everything else in the frame moves. The efficiency problem is real, though. Runway thinks the answer is more compute, better training to squeeze more from every frame, and better algorithms. As Anastasis said in our first conversation, “Naively scaling what we did two years ago won’t solve these problems. I expect the compute needed to train World Models to eventually exceed what LLMs need, and we’ll need real algorithmic progress on top of that.” Good time to be Jensen Huang. The World Model Backbone Anastasis’s bio on X reads “Simple ideas, pursued maximally,” which is also a pretty good summary of Runway’s research strategy. The simple idea at Runway is next-frame prediction. A language model learns to describe the world by predicting the next token in a sequence of tokens. An autoregressive World Model, as Cristóbal put it, learns to simulate the world by predicting the next frame in a sequence of frames. Autoregressive is a fancy word for a relatively simple idea. The model generates one frame at a time, each based on the frames that came before it, the same way an LLM writes one word at a time. Most video models don’t work like this. They generate a whole clip at once, refining it from static into a picture over dozens of steps, which is great if you’re making a ten-second ad and useless if you’re trying to interact. Interactions stop feeling interactive somewhere around half a second of lag. If you want a model that a person, or an agent, or a robot can actually interact with the way humans do with other humans, it needs to produce the next frame quickly, then the next one, then the next, each taking into account whatever just happened in the previous one and in the reaction to it. That’s what Runway did with GWM-1, which it announced last December. It took Gen-4.5, its most recent base video model, and turned it into a model that generates frame by frame, in real time, and can be steered with actions. In its Solaris announcement, asking, “What if software could generate itself, as you interact with it?,” it laid out the general recipe: - Teach the model to generate frames sequentially, - Distill dozens of denoising steps down to a few, - Train the fast model on its own outputs so the quality holds up over a long session. No code underneath, no layout to translate. The image is the application. It treats your clicks and drags the way our video models treats words: not as commands to look up, but as a signal for what to generate next. Nothing about it is decided in advance, so nothing about it has to stay the same… the scene answers to your intentions. It becomes whatever you ask of it. Actions are the important thing. In my (Packy’s) previous essay on World Models, we drew the line between video models and World Models like this: a video model predicts the next frame, P(xt+1 | xt), while a World Model predicts the next state given an action, P(st+1 | st, at). “That at,” we wrote, “is the magic.” Runway’s last month or so of releases can be viewed as a bunch of different at’s. In GWM Worlds 2, the actions are camera movements plus free-form text addressed to anything in the scene. In Runway Characters, the action is audio, a person talking to an avatar that listens and responds. In Solaris, the actions are clicks and drags. In Praxis-1, the actions are robot commands, expressed as where the robot’s hands should go next. There’s a different vocabulary for each, but they all use the same machinery underneath to predict what happens next. I (Packy) have been fascinated by World Models, and really do think they’re some kind of magic, so when I talked to Cristóbal, I wanted to be sure that I understood: “So all of this really just is the same backbone?” “Correct,” he said. “The backbone is frame prediction at scale, which eventually gives rise to a World Model in the same way that next token prediction at scale gives rise to large language models. And the applications are diverse on top.” This is why the company is moving so fast across seemingly disparate applications right now. About 90% of Runway’s research effort goes into pretraining. The belief is that the capabilities get learned there, by watching, and that post-training mostly brings out what the model already knows. Once the base gets good enough, each new application takes far less effort to stand up than the one before it. You don’t, for example, need to build a robotics model from scratch. You can take a model that has already watched an enormous amount of the world and teach it to speak robot. There’s a lot of room left. Runway estimates that Video and World Models are a year or two behind LLMs, which means pretraining is nowhere near saturated. My (Anastasis’) rule of thumb is the number of pixels a model can generate, which is resolution x duration, roughly doubles every few months. I don’t see any fundamental blocker to hour-long generated video within a year or two. Longer videos aren’t simply a matter of doing the same thing, more. A model that has to keep a scene coherent for an hour has to learn persistence. Turn around, and what was behind you is still there. The mug you set down on the counter is still on the counter twenty minutes later, unless somebody moved it, in which case the model should know who moved it, and where. Those are exactly the things today’s models still struggle with; longer context, the company believes, is how you train those problems away. Longer term, the really big bet is that the field moves toward one model that does language, pixels, audio, and actions together. Even in the near-term, though, the World Model backbone that’s used to generate pixels on a computer screen seems to also be able to help robots navigate and interact with the physical world. Teaching Robots with Video Every robotics company with ambitions of deploying at scale runs into the data wall: real-world data is scarce, expensive to collect, and a general-purpose robot needs a lot of it. In Packy’s Robotics piece with Evan, they wrote a whole section titled, “Robotics is Bottlenecked on Data,” on this idea: LLMs had it relatively easy. The entire internet existed as a pre-built training corpus. There is so much more information on the internet than you could ever imagine. Any question you might ask an LLM, the internet has probably asked and answered. The hard part was building architectures that could learn from it all. Robotics has the opposite problem. So robotics companies have to build their own internet equivalent, one demonstration at a time, often with a human in a VR headset puppeteering an arm through the same task over and over again. For jobs with long tails of weird edge cases, like household robots, Runway argues that there’s no practical way to collect all of the data you’d need, let alone enough of it. Video, on the other hand, is effectively limitless. People film and upload more of everyday life every single day than every robot lab on earth could capture through teleoperation. And as generative models get good enough to make video that’s indistinguishable from the real thing, it’s becoming even more limitless. You can start with real scenes and simulate thousands of variations, each with subtle differences. Packy here. In that same Robotics essay, Evan and I wrote a section called “Video Dreams (and Their Limits)” in which we argued that internet video isn’t enough. Video shows you trajectories and timing and the order of steps, but it doesn’t carry mass, force, friction, or stiffness. Two surfaces can look identical and behave completely differently, and the robot finds out the fun way. We cited a study in which people who watched more videos of moonwalking and juggling became much more confident and not much better. Reading lots of books makes it easier to write a good book, we wrote, but watching lots of golf videos doesn’t do much for your golf game. First, hell yeah. This is why I do these co-written essays. Smart people can disagree. General robotics is still unsolved and people much smarter than me are taking many different paths. Second, though, this isn’t the disagreement it appears to be on the surface. As one piece of evidence: Standard Bots is now testing Runway’s Praxis-1 on its RO1 arm. What resolves the seeming disagreement is that Runway isn’t betting on video instead of robot data, but before robot data. Praxis-1, which Runway unveiled at its AI Summit on September 30th, learns mostly from third-person video, built on the same large-scale pretraining as the company’s World Models. From there, Cristóbal explained, “you can take the learnings of large video data and start fine-tuning it on robotics. So you add a bit more egocentric data, and that egocentric data will help the model understand a bit better the situation that it might be in.” Then you fine-tune on the specific robot. How much robot data does that last step take? About 30 minutes of data teaches the model to control a robot body it hadn’t seen in pretraining, and in new environments it can reach strong performance on some tasks with just minutes of data. You still have to swing the club a bit, so to speak, but you need a lot less practice when you have a robust model of how the world works pre-loaded in your brain. As a test, Runway pretrained robot policies on ordinary video and, separately, on teleoperated robot video, using anywhere from ten to a thousand hours of each. Then it fine-tuned them and measured how close the robot got to putting an object where it was supposed to go. The two lines are basically on top of each other: web video worked as well as robot video. The web-video version ended up with a final placement error of 16.1 centimeters versus 16.0 centimeters for the teleoperated robot video version. That’s not nothing; for high-precision tasks, 0.1 cm might as well be a kilometer. But these models are as bad as they’re ever going to be, and teleop data isn’t getting any cheaper to capture. If they’re roughly interchangeable as a starting point, the robotics data problem looks very different than it did in January. But robots’ bodies and forms vary wildly, both from each other and from humans’. How can a model that’s mostly watched humans control a robot’s arm precisely? Part of the answer is in how Runway represents actions. Praxis-1 expresses actions in a standard end-effector format: where the gripper (the “hand” at the end of the arm) should be in 3D space, and how it should be oriented. A person’s hand reaching for a mug in a YouTube video can be described in exactly the same terms. So can a humanoid’s hand, a six-axis industrial arm, or a two-armed setup. While the embodiment changes, what the model is trying to figure out – where should the hand go next – doesn’t really. To that end, Runway’s model is currently being tested on Noble Machines’ bimanual setup, Standard Bots’ six-degree-of-freedom RO1, and Ultra’s mobile robot. In one demo, the same policy works in a controlled studio and in a home kitchen without retraining. It may be helpful to think about the three ways World Models can be used in robotics. The first is generating training data. Take a small set of real demonstrations and create variations with new objects, new lighting, and new layouts with which to train the robots on a bunch of different situations they might not see in real world training very quickly. The second is simulation, or testing robot policies inside the World Model before you put them on real hardware. This is where Runway has published its most rigorous evidence. In February, its robotics team took eight different manipulation policies from the RoboArena benchmark, all running on the same Franka Panda arm, and ran each one inside GWM-Robotics, an early version of Praxis-1. Each policy made its own decisions based on what the World Model showed it. Human graders then scored 1,450 simulated rollouts, more than 16,000 ratings in all, about ten per rollout. The ranking of the eight policies in simulation matched their ranking in the real world with a correlation of 0.95, and the graders correctly picked out both the best and the worst policy. It’s a test of ranking (which policy is better than which), not of absolute success rates, but for a robotics team deciding which of your twenty candidate policies deserves time on your three physical robots, ranking is exactly what you need. The theory here, and across the robotics work Runway is doing, is to do as much in relatively cheap simulation as you can before moving to the more expensive physical world. Runway’s model only needs a starting image, compared with competing approaches that require scanning and reconstructing every scene in 3D, which makes it relatively cheaper than even other simulation approaches. The third way is to use the World Model as the robot’s brain. This is the one that Cristóbal is most excited about. “One of the most compelling use cases that we found for these models in robots is less on the synthetic data part and more on using the video models as the