loading
loading
So many aspects of our lives are going to change over the next twenty years. And one that’s easily in the top five is the question: “What do you do at work all day?” Two massive breakthroughs are on the cusp of happening around this question. The first one is agentic automation: “describe a task that we currently do, and automate it.” The second one is employee empowerment to take initiative. People closest to the problem can increasingly investigate and take action on opportunities without waiting for information and authority to travel through layers of hierarchy. Both of these things are radical changes - for the better! - to what everyone does all day. IF, and this is a big if - organizations are actually able to evolve into this new way of working. A shockingly large component of “what is someone’s day to day job” is constrained by the system of record and decision-making used by their employer. And this in itself is constrained by the database technology they use, and what it makes possible. Not many people think about this second part. Enter Databricks. Outside of people in their sphere of influence (who know them very well), Databricks might be the most under the radar story in Silicon Valley, even though they’ve now surpassed a $7 billion run rate, growing 80% y/y. You’ve certainly heard of them, but not that many people actually know what Databricks does, other than being 1) a data company, and 2) important for agents, although perhaps not sure why. The Databricks story is a wild sequence of events, and a serendipitous example of a product whose time has come. Three years ago, every interaction on Databricks involved a human. Today, the company is seeing more queries from agents than people, and more than 17 million databases are launched by agents every day. This trend will continue. There are four parts to this story: 1) What does Databricks actually do? What is a “Lakehouse” and a “Lakebase”, those words you may have seen on billboards, and why are they an architectural breakthrough that arrived just in time? 2) What is a “system of record”, and why do agents need a new kind of SoR if they’re gonna work inside real companies, make real decisions, and contribute to the company’s knowledge base? 3) What is a firm in the first place? How does that change, if agents outnumber people 1000:1? How do firms differentiate themselves, and create value? 4) What are people’s jobs all day, in this new world? The promise of AI at work is often advertised as, “Ordinary people now have software engineer powers.” But the long-term impact isn’t necessarily “everyone is an engineer.” It might be closer to, “Everyone is a manager.” This is, perhaps, the highest-stakes piece of all. We’re witnessing in real time the great undoing of the management style that built the 20th century, as chronicled in James Burnham’s famous work The Managerial Revolution. Eighty five years ago, Burnham described how information technology and organizational complexity was creating a corporate operating model, where a small class of professional managers centralized the decision-making inside of companies. Today, thanks to a new wave of technology, the craft practitioners at the front lines are regaining responsibility again - and we’re all better for it. This is Databricks, and why it will power the future of the firm. From Spark to Agents Databricks has recently been described as “a data company that caught the AI wave.” But the company was built by AI researchers a decade before the wave arrived. The business is four technology S-curves stacked over nearly two decades – Spark, Lakehouse, Lakebase, and now agents – each of which created the innovation window for the next. To properly appreciate this sequence of unlocking potential, we have to take a minute and talk about database technology. In 2009, Ali Ghodsi was a visiting researcher at UC Berkeley watching internet companies use enormous datasets to predict what people would click, buy, and watch. This was a time when people talked about “Big Data” a lot. The leading framework was Hadoop, which companies like Yahoo, Facebook, Twitter, and LinkedIn used to process data workloads in parallel across thousands of cheap, commodity servers. Before Hadoop, the standard way to process larger workloads was to buy a bigger machine from vendors like Oracle or IBM that cost millions of dollars each, so Hadoop was an attractive alternative. Hadoop’s critical flaw was that each round of a computation saved its results to disk before the next could begin. That made work easier to recover when one of the cluster’s cheap machines failed, which was a sensible choice. But it was painfully slow for iterative workloads like machine learning, where you make predictions, measure the error, update a model, and then repeat. Hadoop treated each round as a new job and reloaded the same data from disk every time, so the cluster spent minutes moving data for every few seconds of improving the model. This problem inspired Matei Zaharia, a PhD student at Berkeley, to write the first version of Apache Spark. Spark built on Hadoop’s innovation of spreading data processing work across cheap machines, but let applications keep frequently reused data in the servers’ much faster memory between rounds instead of loading it from disk each time – without giving up fault tolerance. That made these repetitive workloads up to ten times faster. The lab open-sourced it in 2010, and three years later, Ali, Matei, and five of their fellow researchers founded Databricks to abstract away the complexity of running the distributed infrastructure Spark required – the clusters, the scaling, the security – behind a single platform for the enterprise. Spark sped up data processing, but where that data lived was another problem. Enterprise data in that era occupied two separate worlds. Data warehouses held the structured, trustworthy records that let analysts ask questions of them in SQL and powered every dashboard and quarterly report. Data lakes provided cheap storage for everything else, like documents, images, recordings, and logs. Data lakes were cheap, so teams dumped everything into them; but nothing was checked or structured on the way in. Even though a vast ecosystem of AI open-source tools like TensorFlow and PyTorch grew around the data lake, it was hard to get value from a pile of data you couldn’t trust, which earned them the nickname “data swamps.” Both systems solved a problem, but they created a divide. Engineers and data scientists lived in the lake with Python while analysts lived in the warehouse with SQL. Whenever one side needed the other’s data, someone had to build a pipeline to move it. This left the company with lots of copies, pipelines, permissions, and conflicting definitions. But in the late 2010s, Databricks, Uber, and Netflix each landed on the same fix independently. Their projects (Delta Lake, Hudi, and Iceberg) added a table layer over the files in the lake, giving them warehouse-like guarantees such as all-or-nothing updates, schema management, and versioned history. Instead of moving the data into a separate warehouse for reliability and performance, a company could now run warehouse-style workloads on the same storage where the data already lived. It was the breakthrough that made a new architecture possible. In 2020, Databricks invented the “lakehouse” by merging the data lake and the data warehouse. It combined the cheap, flexible storage of the data lake with the strict reliability and fast query performance of a data warehouse, then made warehouse-grade SQL and machine learning first-class workloads on the same foundation. The cheap storage that held everything became trustworthy enough to serve as the data foundation that engineers, data scientists, and data analysts could finally share. “The best data warehouse is a lakehouse,” went the tagline, and customers seemed to agree. Today, Databricks’ Lakehouse runs at a more than $1.5 billion revenue run-rate, growing over 100% year over year. Amazon CTO Werner Vogels made the case from the customer’s side in a January 2020 blog post. “If you wanted to combine all of this data in a traditional data warehouse without a data lake, it would require a lot of data preparation and export, transform, and load (ETL),” he explains. “You would have to make trade-offs on what to lose and continually change the structure of a rigid system.” Data lakes let you import data in any format, eliminating these trade-offs. The lake was also a powerful foundation for machine learning and AI, which, Vogels wrote, “thrive on large, diverse datasets.” One divide remained. The lakehouse unified the data a company analyzes, but not the data its applications run on. The operational database, which manages real-time transactions for day-to-day business operations, like processing payments or user logins, was still stitched to the lakehouse by a web of fragile pipelines. Compute and storage remained bundled in one monolithic box, and data was trapped in proprietary formats that made migrations one of the scariest technical projects in any organization. So in 2025, Databricks bought Neon, a serverless Postgres company whose databases were already being created mostly by AI agents, and which represented a new architectural category of databases, called lakebase (inspiring Databricks to call the resulting product that integrated Neon into Databricks “Lakebase”). Just as the lakehouse rebuilt the warehouse on top of the data lake, Lakebase applies the same idea to the operational database. It separates storage from compute so data can be stored cheaply in the data lake while compute scales up or down with demand. And because operational databases and the lakehouse share the same underlying storage, database changes automatically become available to analysts, engineers, and agents, without a web of pipelines to build and maintain. Databricks calls this unification of transactions and analytics LTAP, for Lake Transaction/Analytical Processing. This separation makes the database behave more like code. Even petabyte-scale databases can be duplicated in seconds because Lakebase only copies the changes. This lets an agent create an isolated branch, experiment on it, and throw it away when it’s done – a consideration even Databricks didn’t fully appreciate until recently. Lakebase launched last year and has already crossed a $100 million revenue run rate, and agents are its most prolific users. “We think that is probably going to be a bigger market in the long run than the existing database market,” Ali says. This technological unlock has come just in time. For decades, software in a company was built by a small group of engineers and used by everyone else. We are moving to a world where a person declares the outcome they want and the software figures out how to achieve it. Instead of thousands of employees reading and writing from a handful of databases managed by a central team, the company will have thousands of databases that millions of agents read and write to. A System of Record for Agents In June, Ali stood in front of ~30,000 people at the company’s Data + AI Summit and asked the room to raise their hands if artificial general intelligence had arrived. He counted maybe ten percent. “We believe AGI is already here,” he told them, pointing out that today’s frontier models can already solve more than half the questions on Humanity’s Last Exam – 2,500 questions, written by experts in more than a hundred fields, believed to be too hard for any AI to answer. When ChatGPT came out in 2022, it could handle coding tasks that take a person about thirty seconds. Today’s frontier agents complete coding tasks that take a skilled engineer a full work day. Researchers at METR, an independent evaluation lab, have tracked the length of a task an agent can finish on its own across models going back to 2019, and it has doubled roughly every seven months. Recently, progress has been even faster, with similar trends beyond coding. If it holds through the early 2030s, agents will be capable of doing tasks that would take a human years. Yet, inside most companies today, AI still isn’t trusted to reliably update records in Salesforce. “AI does not have an intelligence problem,” Ali explains. “It has a context problem.” Even the smartest human would be all but useless without access to the company’s data, permission to act, feedback, or the knowledge in other employees’ heads. Ali’s answer is what he calls a “system of record for agents.” Maybe you’ve seen a clip of Palantir CEO Alex Karp talking about ontology. “All of the value in the market is going to go to chips and what we call ontology,” he posits. “The ontology allows you to take an LLM, use it, refine it, and impose it on your enterprise.” What that means in practice is usually left as an exercise for the listener. Every company has an ontology – a working model of what exists, how those things relate, what its words mean, and which source governs when two disagree. Parts of it live in databases, dashboards, policies, and documents. But much of it lives only in the heads of experienced employees. It’s also never finished. Contracts change, workarounds become processes, exceptions become rules, and the official record drifts away from how the work is actually done. The same is true of who decides what and of remembering which decisions were right – authority and evaluation are just as unwritten as ontology. People compensate for all of it almost invisibly. It’s the job of the VP of Sales to know that the biggest committed deal has stalled because the champion is quietly interviewing elsewhere, that one rep’s “90%” really means 50% – and to cut the forecast before it reaches the CEO. For agents to act autonomously, much of the ontology, authority, and evaluation that currently lives in people’s heads will have to move into the firm’s system of record. Databricks is betting the lakehouse is the natural home for it because that’s where the rest of the company’s data is stored and governed. “The lakehouse has become the system of record nowadays,” Ali says. “I call it SORA – the System of Record for Agents.” Employees can interact with this system of record through Genie, the natural-language interface Databricks launched in 2024 that lets anyone in the company ask questions of its data and, increasingly, build agents on top of it. Underneath, the record has to span the whole firm and continuously answer four questions, each of which Databricks is working on: What is true? Lakehouse and Lakebase already hold the state. What does it mean? Instead of letting AI loop through all the data in a company hunting for answers every time someone asks a question, Databricks’ Genie Ontology continuously builds a graph of what the organization knows, like how revenue is calculated or what was discussed in the transcript of last week’s standup. An algorithm the engineers dubbed OntoRank (a nod to Google’s PageRank) ranks which knowledge is authoritative by weighing attributes like who created a definition, how widely it’s used, and how recently it changed. In Databricks’ testing, grounding an agent in the company’s ontology raised the accuracy of answers from about 50% to 84.5%. What is allowed? The old access model of a login for every employee held up when every worker was human, but it breaks when employees are spinning up hundreds of agents a day. “You can’t grant access to each proprietary, siloed database to your agents,” Ali explains. Databricks’ Unity Gateway gives every model, agent, and AI tool in the company an identity, a budget, and guardrails, and traces log every action so the company can see what its AI systems are doing, what data they access, and how much they cost. Did it work? Databricks gives companies two ways to build agents. Business teams build theirs in Genie, turning a prompt into a shareable agent that reasons over the company’s data and, increasingly, takes action in applications like Slack or Salesforce. Engineers build more customized agents in Agent Bricks, Databricks’ developer platform for production AI agents. Either way, Databricks lets teams write their own exams – the acceptable results and the mistakes that cannot be made – and gives them tools to optimize each agent against the company’s definition of good for a given task, which is how AI becomes a loop. Every platform shift in enterprise computing has created a new fight over the system of record. IBM dominated the mainframe era. Oracle and SAP became foundations of the client-server enterprise, while Salesforce, Workday, and ServiceNow built the SaaS systems of record for sales, HR, and IT. Agents are the next shift. But which vendor wins is a smaller question than how agents will change the firm itself. The Nature of the Firm In his 1937 article “The Nature of the Firm,” economist Ronald Coase asked why firms exist at all. If markets can coordinate activity through prices and contracts, why bring people together inside companies and subject their work to managerial direction? His answer came down to transaction costs. Finding the right person, negotiating what needs to be done at what price, and monitoring it all take time and money. A firm replaces some of those repeated market transactions with a more open-ended arrangement in which managers direct resources within agreed limits. But organizing work inside a firm is not free either. As a company grows, more information has to travel through more people, and eventually, the cost of organizing the next transaction internally costs as much as buying it from the market. In Coase’s account, the firm expands until those two costs meet. Forty years later, Herbert Simon – one of the few people to win both the Nobel Prize and the Turing Award – observed that centralizing the memory of a business decentralized its decisions. A shared computer record, he wrote, “allows each of the decentralized decision points to take actions independently” because the system automatically reconciles those actions against the shared record. For an airline, a ticket agent in Boston didn’t have to call every other ticket agent before selling the last seat to Chicago. She consulted the record, and when she sold the seat, the record changed for everyone. Centralized state made decentralized action safer because people could coordinate with the record instead of with each other. The relational database extended this principle across the company. Work that once required reports, meetings, and managerial handoffs became a query against the shared record, which let the company coordinate more products, locations, people, and transactions without adding management at the same rate. By the 1990s, the internet let the outside world write to the firm’s record, and an online marketplace could coordinate buyers, sellers, inventory, and payments among millions of strangers. Because both external and internal transaction costs fell, the database era did not make firms uniformly larger or smaller. Instead, it allowed a wider range of firms to exist. Large firms became capable of managing far more complexity, while small firms could rent or buy capabilities that once required entire departments. The same is likely to be true of AI. Inside the firm, agents can reduce coordination costs by automating reporting, search, and routine tasks. Outside it, they can help find and evaluate vendors, negotiate routine terms, and monitor the results. Like the database before it, AI will raise the complexity ceiling of the businesses and markets that can exist. But its more consequential effect may be how it changes the shape of the firm. The typical firm of the twentieth century was pyramid-shaped because hierarchy was its information processing system. Facts moved up through reports and meetings, decisions moved back down, and the map of information flow mirrored the org chart. Management decided what data was worth gathering or sharing, which meant the company’s view of itself was only as good as what its managers had chosen to collect. Databases changed this structure, but only partially. A shared record allowed people to access information directly and act more independently, but usually within procedures that someone else had already designed. For much of the computer era, a relatively small group of managers, operations teams, and engineers defined the company’s schemas, metrics, applications, and workflows – and far more employees used those systems than helped design them. This constrained not only what people could do, but what the organization could know. For example, a support rep who hears the same complaint several times a week knows something the company doesn’t if her manager doesn’t write up a report that captures it. And fixing the problem is another thing entirely. The technical barriers to both – knowing and doing – are now falling at once. As Andrej Karpathy put it in 2023, “The hottest new programming language is English.” Natural-language interfaces allow more domain experts to interrogate a company’s data without waiting for people who know SQL or Python to anticipate or answer every question. At Novo Nordisk, a century-old global healthcare company, scientists used to wait on data engineers to write their queries. Now they ask Genie, “show me all of our obesity data,” and it pulls the relevant trials, insights, and visualizations instantly. And within defined permissions, employees are empowered to create solutions that save hours or even days of work. At the electric vehicle manufacturer Rivian, a finance analyst with no SQL experience used Genie to build a revenue dashboard for the CFO, and a supply chain analyst built an app to track inbound inventory shipments. This doesn’t mean everyone has access to all of the firm’s data or resources without approval. Shared ontology, permissions, and accountability become more important as work becomes more distributed. The system of record centralizes state and guardrails while decentralizing who can ask questions, design workflows, and act. In March, Jack Dorsey wrote a manifesto titled “From Hierarchy to Intelligence” describing how his company Block is beginning to test an extreme version of this idea. “For the first time,” he writes, “a system can maintain a continuously updated model of an entire business and use it to coordinate work in ways that previously required humans relaying information through layers of management.” Ali’s “system of record for agents” is similar to what Jack calls the “world model” of the company – a continuously updated representation of the organization’s operations, decisions, capabilities, and priorities. People at the edges of the company ground the model in reality, while the model supplies the shared context needed to act. “People reach into places the model can’t go yet.” Dorsey explains. “They sense things the model can’t perceive: intuition, opinionated direction, cultural context, trust dynamics, the feeling in a room. They make the calls the model shouldn’t make on its own, especially ethical decisions, novel situations, and high-stakes moments where the cost of being wrong is existential.” The result is a firm that needs fewer layers of management to coordinate it and gives leaders greater visibility into the work, even as it becomes more distributed. Databricks already provides some of the underlying infrastructure for Block. It manages and governs petabytes of Block’s data across Square, Cash App, and TIDAL, and Block used Databricks to ground its AI coding assistant, goose, in the company’s context. For Dorsey, the competitive case for this new architecture is speed. “Companies move fast or slow based on information flow,” he argues. “Hierarchy and middle management impede information flow.” Ali agrees with Dorsey directionally, but with one distinction. “I agree philosophically with Jack Dorsey,” he says. “The question is does that apply to the org chart of humans too?” People conflate two things, he argues: the org chart and the way information moves through the company. “Ten, twenty years ago, these were the same thing.” The only way to learn what was happening was to ask your people, who asked theirs, up and down the tree – but that’s no longer true. “Genie pretty much answers any question I have in any meeting about anything,” he adds. The firm will still need managers, just fewer information routers. Developing people, helping teams decide what’s important, and noticing when someone is unhappy still require human attention. So while the two disagree about how much hierarchy will disappear, both see AI taking over the information-routing work that gave managers immense power over the twentieth-century firm, as companies grew too complex to run without them. The Second Managerial Revolution Eighty five years ago, on the heels of the Great Depression, James Burnham wrote a book called The Managerial Revolution which foresaw a great change in how businesses worked, and even how capitalism itself worked. Coming out of the Depression, one of the prevailing trendy attitudes was that entrepreneurial capitalism had run its course as an economic trend, and some new system was bound to replace it. Socialist collectives were one trendy proposal at the time, nationalization was another. Burnham understood what was going on. He articulated a third kind of thing, not just a corporate structure but an entirely new kind of social order, called the “managerial society.” In this system, the decisive ruling class would be neither the traditional capitalist owners nor the workers, but professional managers, planners, bureaucrats, and executives who operate large corporations and states. Modern institutions, he believed, were too technically complex to be directed personally by individual capitalists. Since no CEO can know everything happening inside a company with a hundred thousand employees, control would gradually migrate from owners and domain experts to administrators. In the second half of the twentieth century, Managerialism scored a near-complete victory. “Most of the executives at Fortune 500 companies are not domain experts at whatever the company does, and they’re certainly not the founders of those companies,” Marc Andreessen comments. “They’re professional managers, and in fact, over the course of their careers, they’ll probably manage many different kinds of businesses – they might work in healthcare for a while, then financial services, and then they’ll work in something else.” But that victory is exhausting itself. The scalability and professional reliability of Burnham’s Managerial Class came at the expense of speed and adaptability, particularly in the face of changing technology. Maybe, Marc suggests, applying agents - on top of a rigorous foundation like Databricks - creates a third model that can offer the best of both worlds. It helps preserve the founder’s vitality and adaptability, while refactoring the managerial work required to scale. Agents will put this way of operating within reach of more founders by automating some of the information routing and coordination work that made the managerial layer necessary, while the firm’s system of record gives the people at the edges the context and authority to act. Elon Musk takes Marc’s idea a step further. Earlier this year, he predicted that companies with “humans in the loop” will lose. Keeping humans in the loop in the age of AI, he argues, will be like using a spreadsheet that requires a person to calculate some of its cells. Elon may be right about the eventual destination – nobody knows how far away the fully autonomous company is, and the forecasts range from years to never. But until then, the irony is that the decline of managerialism may look more like everyone becoming a manager. Not everyone will have direct reports, but more people will be specifying work, delegating execution, inspecting results, and updating the process. The second managerial revolution is not so much the death of management as it is the end of the professional managerial class’s monopoly on it. Software engineers are feeling this acutely today, as the job shifts from artisanally writing code to managing a team of coding agents. An engineer’s day increasingly looks like a manager’s: scope the problem, break it into tasks, delegate them to agents working in parallel, then review the pull requests that come back – approving some and sending others back with notes. As agents become capable across more domains, more knowledge workers will make a similar shift: from performing every step themselves to specifying, delegating, and judging the work. The Novo Nordisk scientists and Rivian supply chain analysts are already directing agents with Genie to do work that used to require a team of engineers or data scientists, freeing those teams to focus on their most strategic work. This is good news for human productivity and salaries because execution and judgment are economic complements: when AI reduces the cost of routine execution, demand rises for the judgment that directs it. The mathematician Alfred North Whitehead observed a century ago: “Civilization advances by extending the number of important operations which we can perform without thinking of them.” The firm advances the same way, becoming more autonomous one loop at a time. Every time a person corrects an agentic loop, the correction should improve the next execution. And as a loop proves it can perform a bounded job reliably, the human moves out of it and to the next unsolved problem, leaving what they have learned in the firm’s shared record. But, as Burnham probably would’ve noticed, for the firm to remain sovereign, it must own its model and everything the people at the edges teach it. The Sovereign Firm Great people will be more valuable than ever in the agentic firm. But great peop
So many aspects of our lives are going to change over the next twenty years. And one that’s easily in the top five is the question: “What do you do at work all day?” Two massive breakthroughs are on the cusp of happening around this question. The first one is agentic automation: “describe a task that we currently do, and automate it.” The second one is employee empowerment to take initiative. People closest to the problem can increasingly investigate and take action on opportunities without waiting for information and authority to travel through layers of hierarchy. Both of these things are radical changes - for the better! - to what everyone does all day. IF, and this is a big if - organizations are actually able to evolve into this new way of working. A shockingly large component of “what is someone’s day to day job” is constrained by the system of record and decision-making used by their employer. And this in itself is constrained by the database technology they use, and what it makes possible. Not many people think about this second part. Enter Databricks. Outside of people in their sphere of influence (who know them very well), Databricks might be the most under the radar story in Silicon Valley, even though they’ve now surpassed a $7 billion run rate, growing 80% y/y. You’ve certainly heard of them, but not that many people actually know what Databricks does, other than being 1) a data company, and 2) important for agents, although perhaps not sure why. The Databricks story is a wild sequence of events, and a serendipitous example of a product whose time has come. Three years ago, every interaction on Databricks involved a human. Today, the company is seeing more queries from agents than people, and more than 17 million databases are launched by agents every day. This trend will continue. There are four parts to this story: 1) What does Databricks actually do? What is a “Lakehouse” and a “Lakebase”, those words you may have seen on billboards, and why are they an architectural breakthrough that arrived just in time? 2) What is a “system of record”, and why do agents need a new kind of SoR if they’re gonna work inside real companies, make real decisions, and contribute to the company’s knowledge base? 3) What is a firm in the first place? How does that change, if agents outnumber people 1000:1? How do firms differentiate themselves, and create value? 4) What are people’s jobs all day, in this new world? The promise of AI at work is often advertised as, “Ordinary people now have software engineer powers.” But the long-term impact isn’t necessarily “everyone is an engineer.” It might be closer to, “Everyone is a manager.” This is, perhaps, the highest-stakes piece of all. We’re witnessing in real time the great undoing of the management style that built the 20th century, as chronicled in James Burnham’s famous work The Managerial Revolution. Eighty five years ago, Burnham described how information technology and organizational complexity was creating a corporate operating model, where a small class of professional managers centralized the decision-making inside of companies. Today, thanks to a new wave of technology, the craft practitioners at the front lines are regaining responsibility again - and we’re all better for it. This is Databricks, and why it will power the future of the firm. From Spark to Agents Databricks has recently been described as “a data company that caught the AI wave.” But the company was built by AI researchers a decade before the wave arrived. The business is four technology S-curves stacked over nearly two decades – Spark, Lakehouse, Lakebase, and now agents – each of which created the innovation window for the next. To properly appreciate this sequence of unlocking potential, we have to take a minute and talk about database technology. In 2009, Ali Ghodsi was a visiting researcher at UC Berkeley watching internet companies use enormous datasets to predict what people would click, buy, and watch. This was a time when people talked about “Big Data” a lot. The leading framework was Hadoop, which companies like Yahoo, Facebook, Twitter, and LinkedIn used to process data workloads in parallel across thousands of cheap, commodity servers. Before Hadoop, the standard way to process larger workloads was to buy a bigger machine from vendors like Oracle or IBM that cost millions of dollars each, so Hadoop was an attractive alternative. Hadoop’s critical flaw was that each round of a computation saved its results to disk before the next could begin. That made work easier to recover when one of the cluster’s cheap machines failed, which was a sensible choice. But it was painfully slow for iterative workloads like machine learning, where you make predictions, measure the error, update a model, and then repeat. Hadoop treated each round as a new job and reloaded the same data from disk every time, so the cluster spent minutes moving data for every few seconds of improving the model. This problem inspired Matei Zaharia, a PhD student at Berkeley, to write the first version of Apache Spark. Spark built on Hadoop’s innovation of spreading data processing work across cheap machines, but let applications keep frequently reused data in the servers’ much faster memory between rounds instead of loading it from disk each time – without giving up fault tolerance. That made these repetitive workloads up to ten times faster. The lab open-sourced it in 2010, and three years later, Ali, Matei, and five of their fellow researchers founded Databricks to abstract away the complexity of running the distributed infrastructure Spark required – the clusters, the scaling, the security – behind a single platform for the enterprise. Spark sped up data processing, but where that data lived was another problem. Enterprise data in that era occupied two separate worlds. Data warehouses held the structured, trustworthy records that let analysts ask questions of them in SQL and powered every dashboard and quarterly report. Data lakes provided cheap storage for everything else, like documents, images, recordings, and logs. Data lakes were cheap, so teams dumped everything into them; but nothing was checked or structured on the way in. Even though a vast ecosystem of AI open-source tools like TensorFlow and PyTorch grew around the data lake, it was hard to get value from a pile of data you couldn’t trust, which earned them the nickname “data swamps.” Both systems solved a problem, but they created a divide. Engineers and data scientists lived in the lake with Python while analysts lived in the warehouse with SQL. Whenever one side needed the other’s data, someone had to build a pipeline to move it. This left the company with lots of copies, pipelines, permissions, and conflicting definitions. But in the late 2010s, Databricks, Uber, and Netflix each landed on the same fix independently. Their projects (Delta Lake, Hudi, and Iceberg) added a table layer over the files in the lake, giving them warehouse-like guarantees such as all-or-nothing updates, schema management, and versioned history. Instead of moving the data into a separate warehouse for reliability and performance, a company could now run warehouse-style workloads on the same storage where the data already lived. It was the breakthrough that made a new architecture possible. In 2020, Databricks invented the “lakehouse” by merging the data lake and the data warehouse. It combined the cheap, flexible storage of the data lake with the strict reliability and fast query performance of a data warehouse, then made warehouse-grade SQL and machine learning first-class workloads on the same foundation. The cheap storage that held everything became trustworthy enough to serve as the data foundation that engineers, data scientists, and data analysts could finally share. “The best data warehouse is a lakehouse,” went the tagline, and customers seemed to agree. Today, Databricks’ Lakehouse runs at a more than $1.5 billion revenue run-rate, growing over 100% year over year. Amazon CTO Werner Vogels made the case from the customer’s side in a January 2020 blog post. “If you wanted to combine all of this data in a traditional data warehouse without a data lake, it would require a lot of data preparation and export, transform, and load (ETL),” he explains. “You would have to make trade-offs on what to lose and continually change the structure of a rigid system.” Data lakes let you import data in any format, eliminating these trade-offs. The lake was also a powerful foundation for machine learning and AI, which, Vogels wrote, “thrive on large, diverse datasets.” One divide remained. The lakehouse unified the data a company analyzes, but not the data its applications run on. The operational database, which manages real-time transactions for day-to-day business operations, like processing payments or user logins, was still stitched to the lakehouse by a web of fragile pipelines. Compute and storage remained bundled in one monolithic box, and data was trapped in proprietary formats that made migrations one of the scariest technical projects in any organization. So in 2025, Databricks bought Neon, a serverless Postgres company whose databases were already being created mostly by AI agents, and which represented a new architectural category of databases, called lakebase (inspiring Databricks to call the resulting product that integrated Neon into Databricks “Lakebase”). Just as the lakehouse rebuilt the warehouse on top of the data lake, Lakebase applies the same idea to the operational database. It separates storage from compute so data can be stored cheaply in the data lake while compute scales up or down with demand. And because operational databases and the lakehouse share the same underlying storage, database changes automatically become available to analysts, engineers, and agents, without a web of pipelines to build and maintain. Databricks calls this unification of transactions and analytics LTAP, for Lake Transaction/Analytical Processing. This separation makes the database behave more like code. Even petabyte-scale databases can be duplicated in seconds because Lakebase only copies the changes. This lets an agent create an isolated branch, experiment on it, and throw it away when it’s done – a consideration even Databricks didn’t fully appreciate until recently. Lakebase launched last year and has already crossed a $100 million revenue run rate, and agents are its most prolific users. “We think that is probably going to be a bigger market in the long run than the existing database market,” Ali says. This technological unlock has come just in time. For decades, software in a company was built by a small group of engineers and used by everyone else. We are moving to a world where a person declares the outcome they want and the software figures out how to achieve it. Instead of thousands of employees reading and writing from a handful of databases managed by a central team, the company will have thousands of databases that millions of agents read and write to. A System of Record for Agents In June, Ali stood in front of ~30,000 people at the company’s Data + AI Summit and asked the room to raise their hands if artificial general intelligence had arrived. He counted maybe ten percent. “We believe AGI is already here,” he told them, pointing out that today’s frontier models can already solve more than half the questions on Humanity’s Last Exam – 2,500 questions, written by experts in more than a hundred fields, believed to be too hard for any AI to answer. When ChatGPT came out in 2022, it could handle coding tasks that take a person about thirty seconds. Today’s frontier agents complete coding tasks that take a skilled engineer a full work day. Researchers at METR, an independent evaluation lab, have tracked the length of a task an agent can finish on its own across models going back to 2019, and it has doubled roughly every seven months. Recen
document fetched 2026-10-28 02:53 · canonical https://a16z.news/p/databricks-and-the-future-of-the