loading
loading
“Everyone is entitled to his own opinion, but not to his own facts.” - Daniel Patrick Moynihan (senator, sociologist, and a person who never once had to reconcile two different NRR numbers in a board meeting) Self-serve analytics finally works. Hurray! Agents plus the MotherDuck MCP mean anyone can query the warehouse without ever having written a lick of SQL in their life. And while that’s all incredibly exciting to me and to everyone else who feels this huge shift in velocity compared to 18 months ago, it’s also… a little terrifying. As Chief of Staff at MotherDuck, I own our operating plan and all the reporting and rituals related to how we’re executing against that plan. My job is to help MotherDuck understand itself, and make sure everyone is on the same page so that we can make the right decisions for the future. The problem is that self-service analytics, when left to its own devices, doesn’t tend to converge on a single view of the truth. It fragments into a dozen private versions of it, all with a slight bend towards how the end user perceives the world. We are LIVING this at MotherDuck right now. Everyone here is spinning up Dives and Flights. A salesperson who has never written a SELECT statement in their life can open the MotherDuck MCP server and build a view of net revenue retention (NRR) joined against external demographic data across their book of accounts, without ever talking to the data team. That is incredible. It’s a world that people have been dreaming about for years, and there’s a graveyard of startups from the past two decades of people who have tried and failed to make it happen. I think Ian Macomber from Ramp said it really well on a recent podcast: In this brave new world, the scarce resource is no longer access to the data. It's consensus. Alternative facts Here's the thing about that salesperson. If they don't compute NRR the way the rest of us agreed to compute it (same grain, same account exclusions, same treatment of partner revenue) they walk into our weekly business review with a number that doesn't match mine. Best case, two people who don’t know how to read or write SQL burn twenty minutes doing forensic accounting. Worse case, their dashboard looks so cool that three other reps clone it, and before anyone notices, the entire revenue org is running its book off the wrong definition of retention. Now multiply that by every person and every metric in the company. Terrifying. In the pre-AI world, the data team was the gatekeeper. You didn't make your own dashboard, you asked Kenan to make you one, and you complained to your boss about how slow it was. But, hidden in that bottleneck was doing something we all took for granted: it meant someone was full-time responsible for being right. The data team obsessed over modeling, freshness, and lineage, and the rest of us relaxed a little, knowing there was a single throat to choke when a number was wrong (and it wasn’t ours). The self-serve revolution removed that gatekeeper overnight. Everyone is feeling the velocity gain, and getting totally addicted to it. Fewer people are noticing consensus degrading. People trust the dashboard they built more than the one someone handed them… which is human and reasonable and also exactly the problem. Left alone, self-serve doesn't drift toward the truth. It fragments into everyone privately trusting their own reality. To be clear: I don't think that the state we’re in is a net-bad thing, and even if I did, it’s too late to put the genie back in the bottle. The question I’ve had to reoriented myself around is: how do we get all these newly agent-empowered people to land on the same version of reality? The answer is context. What's a human actually for? I felt this problem emerging almost immediately when we started duck-fooding the early version of Dives at MotherDuck. In a pre-AI world, I’m not sure I, a non-technical person, could have owned this on my own. In a post-AI world, this is entirely possible. Here’s how we approached it. I started with getting a small tiger team together to agree on definitions for our most common metrics. We bootstrapped it the lazy way: we ran a pretty basic prompt on top of the warehouse and asked it to build a “big dumb document”, basically a data dictionary reverse-engineered from what the LLM could see. It did a decent job. An agent can introspect your schema: table and column names, types, foreign-key relationships, whatever comments you were disciplined enough to leave behind. From that it reconstructs the stuff everyone already agrees on. What's the canonical revenue table? What does a contraction event look like? I would say the big dumb document came back about 50% right. The other 50% is the hard part, and where a human needs to get involved. No amount of column-name introspection tells you all of the nuances of your business. Let’s take what counts as “churn” in a usage-based revenue business as an example. Is it a canceled contract? 30 consecutive days of zero compute? But what if they come back on day 31? For the complicated or controversial metrics, write down all of the nuances and complexities onto paper, and lay out the different ways you could calculate the metric depending on what you care about. Go pressure test those with one or two people who interact with the inputs to that metric every day. For churn, that was customer success (affectionately called Hatchery at MotherDuck) and sales. Get them to poke holes in your assumptions and call out things you missed. You probably have decent instincts about the churn patterns in your customer base, so overlay those on your actual data to see if the patterns you care about are getting captured in this definition. For something like churn, you can even go so far as to pull up the actual list of customers who counted as “churned” in the last week or month and ask yourself whether those are indeed churned customers in the spirit of the definition. If you see someone who actually is just experiencing some seasonality and doesn’t really feel like a lost customer, great, use that to tweak your definition. Do this 2-3 times max. You could get caught in the edge cases forever, and you don’t want perfect to get in the way of good enough. Use what you learn to come to the table with a clear recommendation and a sample run of the data with the new definition. People tend to give better feedback when they have something concrete to react to vs. asked about a topic generally, with no guidelines. This isn't a post on how to reach exec alignment (my quick tips: be discerning about which metrics you govern centrally, take input but understand the definition will not be decided democratically, and metrics trees with owners are a genuinely useful tool for figuring out what's worth fighting over). But, it’s a call to action for you, the non-technical human, to focus your time and energy on these sorts of things, and let the LLM handle everything downstream. Turning “here's how we define NRR, at monthly grain” into correct, performant SQL is precisely the kind of work LLMs are great at. The highest-leverage thing that a human (technical or not) contributes to this process is not writing the query. It's manufacturing the consensus that the query executes against. You are responsible for the what. The LLM handles the how. Where should context live A friend I used to work with at dbt is now on the data team at an AI company where everyone internally builds agents. His team did all the beautiful analytics engineering work that dbt pioneered, modeling every metric cleanly in their BI tool. They're confident that if your agent connects there, it comes out with the right answer. I’ve heard data teams at other companies say something similar, whether it’s the BI tool, a data catalog (or whatever we’re calling the third party context layer these days), or something else. The problem is that most people don't connect through some intermediary. LLMs want to take the path of least resistance, which means 9 times out of 10 it’s just going to query the data warehouse directly. It’s hard to enforce making a stop along the way, especially if not all people (or agents) know that stop exists. My friend knows this, and he says it’s one of the biggest things keeping him up at night. This is why I, a non-technical human, have become deeply convinced that context has to live as close to the raw data as possible. If your definitions sit in a modeling layer up the stack, they only protect the agents that bother to route through that layer. And in an AI-native company, you cannot enforce that routing. A context layer that can be easily bypassed isn't actually a context layer. At that point, it’s just a museum artifact. Guides! All of this is why I'm SO excited about Guides. A Guide is a plain markdown document, stored in the MotherDuck warehouse, that agents read before they write SQL. It captures everything your schema can't tell an agent on its own, like how you define a metric, which tables to join, which columns to exclude and why. Some highlights that I’m especially excited about: It's agent-native. Guides load automatically through the MotherDuck MCP. On connect, the agent gets an index of guide paths. It sees that core-metrics/revenue/revenue-retention.md is the one it wants, reads that single file, and only then generates SQL. By the time it touches your data, it already knows to exclude the right accounts and compute at the right grain. You write it in plain markdown. Author it in the UI, dictate it from an agent chat, or push it from your data repo in CI. It reads like a short doc: purpose, key tables, the canonical definition, join patterns, known pitfalls. The workflow I'd recommend is what we did internally: let an agent bootstrap the V1 off your schema to get something that’s 50% there, then spend your human time on the 50% that requires discernment and (dare I say it) taste. Org-wide or personal. Admins publish the definitions everyone shares. A power user who wants their own spin can layer a personal Guide on top, but the org version supersedes it. You give people flexibility to define things that aren’t centrally governed (usually things that are very domain specific, or changing quickly) and enforce the most important metrics that everyone needs to have right. Versioned and auditable. Every save is a new version with a change comment. When a number moves next month and someone asks why, you can point at which definition was live when the query ran. There's a real push-and-pull in AI-native circles about determinism versus letting an agent be free and creative. My current mental model is that you want to be deterministic about the what (the definition, the filters, the grain) and let the agent do its thing on the how (ex. whether it uses a CTE or a window function, how it orders the joins…even if you have no clue what all that means). Don’t be afraid of the fog bank Some people have described AI as a fog bank rolling in on a long stretch of highway: you used to be able to see for miles, and now you can't see more than a few feet ahead. As tempting as it is to slow down and wait for it to clear, I really don’t think that’s an option in this environment. Like I said, regardless of whether you find this new reality of self-serve analytics exciting or terrifying, you aren’t going to be able to put the genie back in the bottle. I think taking control of the context layer is one of the most high leverage things a non-technical domain expert can do right now. We’re in a moment where something that may have felt intimidating 18 months ago, is (1) totally possible for a nontechnical person to own, and (2) can actually be codified into workflows that allow everyone to function better. My advice: just try it. You’ll be surprised at what you can accomplish with the tools we have today. And if you want a data infrastructure solution that brings this all together in one platform, check out MotherDuck! We’ve got a free trial you can sign up for here. Table of contents Read Next
“Everyone is entitled to his own opinion, but not to his own facts.” - Daniel Patrick Moynihan (senator, sociologist, and a person who never once had to reconcile two different NRR numbers in a board meeting) Self-serve analytics finally works. Hurray! Agents plus the MotherDuck MCP mean anyone can query the warehouse without ever having written a lick of SQL in their life. And while that’s all incredibly exciting to me and to everyone else who feels this huge shift in velocity compared to 18 months ago, it’s also… a little terrifying. As Chief of Staff at MotherDuck, I own our operating plan and all the reporting and rituals related to how we’re executing against that plan. My job is to help MotherDuck understand itself, and make sure everyone is on the same page so that we can make the right decisions for the future. The problem is that self-service analytics, when left to its own devices, doesn’t tend to converge on a single view of the truth. It fragments into a dozen private versions of it, all with a slight bend towards how the end user perceives the world. We are LIVING this at MotherDuck right now. Everyone here is spinning up Dives and Flights. A salesperson who has never written a SELECT statement in their life can open the MotherDuck MCP server and build a view of net revenue retention (NRR) joined against external demographic data across their book of accounts, without ever talking to the data team. That is incredible. It’s a world that people have been dreaming about for years, and there’s a graveyard of startups from the past two decades of people who have tried and failed to make it happen. I think Ian Macomber from Ramp said it really well on a recent podcast: In this brave new world, the scarce resource is no longer access to the data. It's consensus. Alternative facts Here's the thing about that salesperson. If they don't compute NRR the way the rest of us agreed to compute it (same grain, same account exclusions, same treatment of partner revenue) they walk into our weekly business review with a number that doesn't match mine. Best case, two people who don’t know how to read or write SQL burn twenty minutes doing forensic accounting. Worse case, their dashboard looks so cool that three other reps clone it, and before anyone notices, the entire revenue org is running its book off the wrong definition of retention. Now multiply that by every person and every metric in the company. Terrifying. In the pre-AI world, the data team was the gatekeeper. You didn't make your own dashboard, you asked Kenan to make you one, and you complained to your boss about how slow it was. But, hidden in that bottleneck was doing something we all took for granted: it meant someone was full-time responsible for being right. The data team obsessed over modeling, freshness, and lineage, and the rest of us relaxed a little, knowing there was a single throat to choke when a number was wrong (and it wasn’t ours). The self-serve revolution removed that gatekeeper overnight. Everyone is feeling the velocity gain, and getting totally addicted to it. Fewer people are noticing consensus degrading. People trust the dashboard they built more than the one someone handed them… which is human and reasonable and also exactly the problem. Left alone, self-serve doesn't drift toward the truth. It fragments into everyone privately trusting their own reality. To be clear: I don't think that the state we’re in is a net-bad thing, and even if I did, it’s too late to put the genie back in the bottle. The question I’ve had to reoriented myself around is: how do we get all these newly agent-empowered people to land on the same version of reality? The answer is context. What's a human actually for? I felt this problem emerging almost immediately when we started duck-fooding the early version of Dives at MotherDuck. In a pre-AI world, I’m not sure I, a non-technical person, could have owned this on my own. In a post-AI world, this is entirely possible. Here’s how we approached it. I started with getting a small tiger team together to agree on definitions for our most common metrics. We bootstrapped it the lazy way: we ran a pretty basic prompt on top of the warehouse and asked it to build a “big dumb document”, basically a data dictionary reverse-engineered from what the LLM could see. It did a decent job. An agent can introspect your schema: table and column names, types, foreign-key relationships, whatever comments you were disciplined enough to leave behind. From that it reconstructs the stuff everyone already agrees on. What's the canonical revenue table? What does a contraction event look like? I would say the big dumb document came back about 50% right. The other 50% is the hard part, and where a human needs to get involved. No amount of column-name introspection tells you all of the nuances of your business. Let’s take what counts as “churn” in a usage-based revenue business as an example. Is it a canceled contract? 30 consecutive days of zero compute? But what if they come back on day 31? For the complicated or controversial metrics, write down all of the nuances and complexities onto paper, and lay out the different ways you could calculate the metric depending on what you care about. Go pressure test those with one or two people who interact with the inputs to that metric every day. For churn, that was customer success (affectionately called Hatchery at MotherDuck) and sales. Get them to poke holes in your assumptions and call out things you missed. You probably have decent instincts about the churn patterns in your customer base, so overlay those on your actual data to see if the patterns you care about are getting captured in this definition. For something like churn, you can even go so far as to pull up the actual list of customers who counted as “churned” in the last week or month and ask yourself whether those are indeed churned customers in the spirit of the definition. If you see someone who actually is just experiencing some seasonality and doesn’t really feel like a lost customer, great, use that to tweak your definition. Do this 2-3 times max. You could get caught in the edge cases forever, and you don’t want perfect to get in the way of good enough. Use what you learn to come to the table with a clear recommendation and a sample run of the data with the new definition. People tend to give better feedback when they have something concrete to react to vs. asked about a topic generally, with no guidelines. This isn't a post on how to reach exec alignment (my quick tips: be discerning about which metrics you govern centrally, take input but understand the definition will not be decided democratically, and metrics trees with owners are a genuinely useful tool for figuring out what's worth fighting over). But, it’s a call to action for you, the non-technical human, to focus your time and energy on these sorts of things, and let the LLM handle everything downstream. Turning “here's how we define NRR, at monthly grain” into correct, performant SQL is precisely the kind of work LLMs are great at. The highest-leverage thing that a human (technical or not) contributes to this process is not writing the query. It's manufacturing the consensus that the query executes against. You are responsible for the what. The LLM handles the how. Where should context live A friend I used to work with at dbt is now on the data team at an AI company where everyone internally builds agents. His team did all the beautiful analytics engineering work that dbt pioneered, modeling every metric cleanly in their BI tool. They're confident that if your agent connects there, it comes out with the right answer. I’ve heard data teams at other companies say something similar, whether it’s the BI tool, a data catalog (or whatever we’re calling the third party context layer these days), or something else. The problem is that most people don't connect through some intermediary. LLMs want to take the path of least resistance, which means 9 times out of 10 it’s just going to query the data warehouse directly. It’s hard to enforce making a stop along the way, especially if not all people (or agents) know that stop exists. My friend knows this, and he says it’s one of the biggest things keeping him up at night. This is why I, a non-technical human, have become deeply convinced that context has to live as close to the raw data as possible. If your definitions sit in a modeling layer up the stack, they only protect the agents that bother to route through that layer. And in an AI-native company, you cannot enforce that routing. A context layer that can be easily bypassed isn't actually a context layer. At that point, it’s just a museum artifact. Guides! All of this is why I'm SO excited about Guides. A Guide is a plain markdown document, stored in the MotherDuck warehouse, that agents read before they write SQL. It captures everything your schema can't tell an agent on its own, like how you define a metric, which tables to join, which columns to exclude and why. Some highlights that I’m especially excited about: It's agent-native. Guides load automatically through the MotherDuck MCP. On connect, the agent gets an index of guide paths. It sees that core-metrics/revenue/revenue-retention.md is the one it wants, reads that single file, and only then generates SQL. By the time it touches your data, it already knows to exclude the right accounts and compute at the right grain. You write it in plain markdown. Author it in the UI, dictate it from an agent chat, or push it from your data repo in CI. It reads like a short doc: purpose, key tables, the canonical definition, join patterns, known pitfalls. The workflow I'd recommend is what we did internally: let an agent bootstrap the V1 off your schema to get something that’s 50% there, then spend your human time on the 50% that requires discernment and (dare I say it) taste. Org-wide or personal. Admins publish the definitions everyone shares. A power user who wants their own spin can layer a personal Guide on top, but the org version supersedes it. You give people flexibility to define things that aren’t centrally governed (usually things that are very domain specific, or changing quickly) and enforce the most important metrics that everyone needs to have right. Versioned and auditable. Every save is a new version with a change comment. When a number moves next month and someone asks why, you can point at which definition was live when the query ran. There's a real push-and-pull in AI-native circles about determinism versus letting an agent be free and creative. My current mental model is that you want to be deterministic about the what (the definition, the filters, the grain) and let the agent do its thing on the how (ex. whether it uses a CTE or a window function, how it orders the joins…even if you have no clue what all that means). Don’t be afraid of the fog bank Some people have described AI as a fog bank rolling in on a long stretch of highway: you used to be able to see for miles, and now you can't see more than a few feet ahead. As tempting as it is to slow down and wait for it to clear, I really don’t think that’s an option in this environment. Like I said, regardless of whether you find this new reality of self-serve analytics exciting or terrifying, you aren’t going to be able to put the genie back in the bottle. I think taking control of the context layer is one of the most high leverage things a non-technical domain expert can do right now. We’re in a moment where something that may have felt intimidating 18 months ago, is (1) totally possible for a nontechnical person to own, and (2) can actually be codified into workflows that allow everyone to function better. My advice: just try it. You’ll be surprised at what you can accomplish with the tools we have today. And if you want a data infrastructure solution that brings this all together in one platform, check out MotherDuck!
document fetched 2026-10-24 10:06