Taking a look under your agent’s hood

extract

Datadog is an observability and security platform for cloud applications, providing real-time monitoring for infrastructure, distributed systems, and AI agent behavior. Connect with Yanbing on LinkedIn and learn more on the Datadog website. Congrats to Great Answer badge winner insertusernamehere for winning the badge on their answer to How can I get a character array from a string?. TRANSCRIPT Ryan Donovan: Hello and welcome to the Stack Overflow podcast, a place to talk all things software and technology. I'm your host, Ryan Donovan. And today we're talking about agents. What are they building in there? And how to find out. My guest for that is Yanbing Li, who is the chief product officer at Datadog. So, welcome to the show, Yanbing. Yanbing Li: Thank you for having me, Ryan. Ryan: My pleasure, my pleasure. Before we figure out what those pesky agents are doing, can you give us a little flyover of how you got into software and technology? Yanbing: I feel like I've been in software forever, for the most part of my life. You know, I would characterize my education as, you know, I've studied until there is no more school to go to. So I have a PhD in electrical and computer engineering. Then I've been an engineer ever since, except for at Datadog, I'm far more focused on product than combinations of product and engineering. And my introduction to observability started with my time at Google. This is seven years ago when I was leading the observability effort at Google, both for powering Google's infrastructure and services, as well as being built as product as part of the Google Cloud capability. So that was my first introduction to observability. Then I had an interesting detour into everything AI, where I was building self-driving trucks at a company called Aurora. And Aurora today is probably still the, not just the first, but the only company that's operating driverless trucks commercially on U.S. public roads. So that was kind of the experience of absolutely needing to understand everything that AI agent is doing. You know, granted, self-driving started before the ChatGPT LLM days, but there is definitely a lot of parallels in thinking about what it takes to fully unleash your agent into the wild. Ryan: Observability is one of those things that's really ramped up a lot since, you know... Datadog used to be a, you know, metrics and dashboard company, but now it's fully part of the observability space. And everybody's trying to figure out, how do you apply observability to agents? How are you thinking about keeping an eye on agents when they're running? Yanbing: As you said, observability has existed for a long time for a really complex distributed system. You know, that part remains the same. But with AI agent, there are certain things that's new and different. You know, first of all, you know, starting with AI models playing an instrumental role in bringing reasoning and intelligence into AI agent. You know, now we deal with the non-deterministic nature of AI models. I think that tends to be the first thing that when we talk about AI observability, we started with LLM observability, really to understand what the LLM is doing. But now, that is quickly expanding from just observing the AI models and the interaction with the AI models to really understand the AI agent as a whole. If we think about AI agent, it's still fundamentally a distributed system. It's just, this distributed system is now powered by intelligent models underneath. So, there are certain things remain the same. We have to understand these new nuances that's introduced by AI. Ryan: Yeah. The observability of LLMs is still a sort of ongoing research project, as I understand it. To get a sort of greater detail into explainability. Is that right? Yanbing: There's certainly certain things at the research frontier, but I also think there are lots of things that's being in production. The fact that we do rely on AI agents in our day to day use, in some of the enterprise use cases, is an indication that LLM observability, or agent observability, in combination with other things, is giving us levels of comfort to rely on these AI agents. And I do think it's a journey because, you know, I work with lots of the enterprises. They're still in this early phase of AI development. Can I trust my AI agent not to stay up, but also to do the right thing? I've seen lots of successful agents. But, you know, when I speak to our customers, many more of them are still in the process of building that. And given the rapid landscape of AI, I'm sure this, how we do it, is going to be evolving continuously. Ryan: The LLM was, you give it an input, it gives a response. And now there's multi-step turn taking, there's tool use. What are the pieces that you have to observe in an AI agent? Yanbing: I think you said it really well. You know, certainly we've started with understanding prompts and LLM output. But because agents are becoming this distributed system in nature, so it's becoming a lot more complicated. You know, we do understand, you know, the reasoning, the two calls, the prompts and the responses, and we need to understand the behavior. We need to understand why it's making certain decisions. And we need to also broadly understand, what's the implication across the applications and outcome? So, I think we are definitely evolving from simply observing just the software stage, software health, to truly observing software behavior. Ryan: The comparison to network software is interesting. I've been saying that AI agents are sort of speed running the microservices path. And for the network software, observability has kind of had 3 or 4 pillars. The logging metrics, traces, and events. Do those traditional pillars apply to the AI software? Yanbing: I think they do still apply, but there is also more things that is definitely emerging in the realm of agent observability. We're observing additional characteristics. So the traditional pillar still is important from what I see in our customer base. We've seen AI generate tremendous demand for observability, and lots of that is through the bread and butter of the microservices, the distributed system, observability of metrics, traces, logs. That is still very, very foundational. But then you added this additional layers of, uniquely understand model interactions and agent behavior. So I think it's actually, the need for observability is growing in both of those. I was actually looking at some of our own agent observability product statistics, just to see what kind of errors it catches. For example, it catches a big chunk of the error in hitting rate limit of API calls, where data access. This is nothing to do with agent per se. It's a problem exists, but it gets manifested even more pronouncedly in agents given the... Increase the speed and scale at data access. So still, a lot of things remain the same. But then there are these new primitives as we've been discussing as well. Ryan: Yeah, yeah. With AI agents, I feel like you can automate more failures. Yanbing: You know, certainly there is token limit, but it is often just the regular rate limit. You know, a lot of the software, they were designed for human or programmatic access. And, you know, agents are exposing lots of rate limiting. You know, we also constantly receive demand from our customers of raising rate limit in our system. I think this is something that's old, but is new in a different way. Ryan: You know, when you think about observing agents outside of the standard failures, you could have incorrect tool calls, you could have PII exposures, you can have all these things that are kind of outside of most, you know, network systems that aren't AI. Are you thinking about applying observability to those sort of issues? Yanbing: Yes, we definitely are applying observability to those issues. You know, when we started doing agent observability, the initial focus was much more on just LLM call itself. You know, the prompt and response. And clearly this has now expanded into the full trajectories of, you know, the calls and the loops. And so, given the sophistication of agent has increased tremendously. And so, those are all the things we observe. I also think simply having telemetry and having, you know, those signals is also not sufficient. Because, you know, how that translates into the understanding of what is not working, and where do I need to improve? And if I think about the contrast between the traditional system and an LLM system is, the traditional cloud software system, or... You know, it requires production feedback loop, but you deploy, you trust in production fairly, you observe everything and you go back and you fix bugs or fixed issues as they arise. But for AI agent, I think there is an argument for much more continuous monitoring, because a lot of the things that can be verified in a traditional system simply cannot be exhaustively verified in a AI agent system. So, I think that feedback loop is much, much more important. In traditional software, it tends to be bug fixing or understanding production load, or sometimes understanding customer behaviors and how you improve your product. But in the world of AI, all of those things applies, but there is more of a... No matter how much we test, how much we evaluate during that build phase, the real world always surprise us in many, many different ways. So I do think that continuous feedback loop and that rapid iteration is becoming much more crucial in the AI world. And this is where we see in AI, the agent observability is no longer just monitoring production, but also how that becomes a flywheel that help you iterate the upstream development. So you're constantly improving the outcome of your agent. And again, also, this thing is also, especially now with AI coming to development. So there is an expectation of that loop becoming a much faster loop as well. So, that is something that guided us to think about, you know, agent observability, not just observing the full production traces, you know, trajectories and two calls and token usage and all of that. Evaluate, you know, both through those online evaluations of my AI agent, how it's doing, and use that to feed back upstream, going back to that build phase, so that we're constantly improving AI. Ryan: Obviously, a lot of people are talking about loops and agents. And, you know, feeding back production data to the build phase of the software. Like you said. Are you thinking about automating that? You know, mining the signals from telemetry, giving it as a sort of direct Jira ticket or whatever for the builders, and then feeding that to an agent to automatically heal the software? Yanbing: So, certainly software development has been proven to be kind of still the dominant AI use case on the market, where people are spending most of their investment. But we're definitely seeing, with AI accelerating software development, that you can deploy, you know, many, many parallel agents to do software development, that shipping software becomes the bottleneck, and shipping software, and iterating software. And we think about the software development cycle. Besides writing the PR, then you go through testing code review, and testing and merging, and pushing it to deployment, and maybe in a staging environment, and then roll out. I think each of these stage can be automated, as well as maybe the whole loop can be automated. I think this is still an evolving landscape. We're trying to do this in our own SDLC. We're also trying to see, what is the product angle? So for example, you know, autonomous testing. What's the new set of code? What kind of new user behavior or product behavior this may be exposing. And how can you autonomously generate test cases so that doesn't become the bottleneck? And obviously there is also the outer loop of how you turn that SDLC, that more sequential process, into a more autonomous loop that can be paralleled in many different stages. This is definitely something interesting to us. We're trying to do this for ourselves. And potentially think about how to do that for our customers as well. And I see a lot of advanced development organizations is all trying to do something very similar. Ryan: There's a lot of data produced by observability, right? A lot of telemetry. Are you finding ways to use AI to sort of automatically flag those unknown use cases, or produce new tests, or things like that? Yanbing: This is definitely a new AI frontier that we're invested in. And so, you know, traditionally we've always had testing that, you know, you have certain type of, either test cases based on previous known production use cases, or synthetic use cases. And the key is, how do you... Let's say you're using AI to write new software. How do you understand the intention and the goal you want to achieve in those new functionality? And how to turn those goals into new synthetic tests that you can run during your software development before it goes into production? So, there is how much you test before you push the software. But as you push the software, how do you bring that production feedback quickly, so that you have that feedback loop working before you start a broad rollout of the software? I think we've been doing lots of those already in today's software development. It's just going to be, we're using AI to make lots of those things fully, automatically done, rather than done by human, and also using AI to create bigger loops, to again, automate from a workflow point of view. Ryan: The workflow is an interesting one, because I think, most developers encounter agents through their coding agent. Is there a case for applying observability to one's own local coding agent process? Yanbing: Shipping software is about code meeting production. You know, you need to deeply understand what that implication is. And we've traditionally do that in... The process that defines today's SDLC. And so, as I mentioned, there is this way of bringing, first of all, synthetic tests early on, and also bringing production signals as you manage your deployment. How you autonomously apply feature flags to get your rollout. And if you get good signals, you expand. If you don't, you roll back. So there is, you know, the autonomous loop that can be created. And we are also in the process of experimenting with a completely local agent, because, you know, developers love their laptops and everything local. And we are actually building a local experience that allows the production context to be available. You know, through Datadog and CPS and et cetera. The boundary for observability, I think, is shifting. Observability used to be squarely, is the thing you hand over to, whether it's the SRE team or the people carrying that production responsibility, and understand. And I think with this new AI based software development, that boundary is kind of blurry. For example, the SREs can start to invest in code and fix code. Because, you know, again, AI destroyed that knowledge boundary. So if I'm running production with AI, the way I understand what's going on with my productions or dependencies on all the different services I don't own, you know, the SREs have the ability to do that and potentially even make upstream service change if their organization permits that. And on the other hand, I think developers are all using production context for their development. You know, through easy access to production telemetry through MCP. And we definitely see a lot of our AI native customers, you know, operate in this way much, much more pronouncedly. Ryan: The boundaries are blurring. Like, any time you start automating parts of your process or your software, you need some sort of signal that it's going correctly, right? That's the domain of observability. Yanbing: So, I think SDLC is being rewritten, and it's definitely an interesting time. And I think this, you know, coming back to where we started, with AI models, we definitely see the same thing. You know, when I was building self-driving trucks, so, we spent a lot of money on training. We spent lots of money on offline... Testing is essentially offline evaluation. Obviously with self-driving, this is L4 level, basically driverless, so you have uncompromised expectation for coverage, for thoroughness or for, you know, certain behaviors absolutely needs to be guaranteed and verified. So a lot of that is done in an offline fashion. And then you deploy the software onto the trucks, you drive around, you see things, you're running in a development mode, or you're running in a production mode. So this is definitely a closed loop exercise. I think we definitely see the same with LLM development. So when we think about agent observability, it's not just about observing production behavior, you know, getting all these telemetries, but, you know, constantly running evaluation to understand, is your LLM behaving correctly? Is it providing the right quality, the right levels of safety? And we also do the same, you know, bring similar technology to do so during the development phase. Development phase, as you, you know, whether you're making model selections or you're building models, you're tweaking them, whether you're understanding what's the interaction between prompts and models and tools, whether you're thinking of, I'm running a certain set of experiments. You know, all of those things is, we see there is the opportunity for those type of tool to be connected. Because there's lots of things, I know you, in your previous podcast, right? How you talked about LLM as a judge. This is a very popular way for us to evaluate LLM behavior. But why do you do it in production this way and do it in development in a different way? So, when we think about agent observability, we are definitely seeing the need of building capabilities that can monitor in production, but also can provide a similar set of capability to help you evaluate, experiment, and ship your agent product. And I think other vendors probably are thinking the same. So it's definitely no longer just for the production business. The development side is very closely related. Ryan: That's something everybody is looking at and trying to figure out where it goes, how you fit that into a system. Do you think that that sort of LLM eval system belongs as part of the observability suite? Yanbing: Certainly from the Datadog point of view, you know, we have observability probably written on our forehead. But I do think a lot of the problems go beyond the previous narrow sense of observability. So rather than saying, a particular category is evaluation, observability, in the narrow sense, it may not be. But from... If I were an engineer, I'm an AI engineer building my next agent. Do you want those things to be connected? I see there's tons of benefit for those things to be connected. It's also just like, you know, we talk about agent observability, or LLM observability, is also not sitting on its own island, because all of those things are sitting inside your system. They access your databases, it goes to your lake houses, API calls of other dependency service. It does the same thing. Except with this, you know, AI magic, you know, besides that. So we also believe, you know, agent observability is not isolated thing. AI agent itself is a distributed system itself, and is related to other broader distributed system. We believe agent observability is part of the broader, you know, system service observability story. And we're also, as I mentioned, whether it's in just general software development or AI agent development, this closeness between build and observe is coming together. So rather than arguing categorically, you know, is this observability or not? We certainly think there is tremendous value of bringing products that can straddle these two different aspects of software development. Ryan: You know, we talked about earlier, like, the amount of telemetry is pretty massive, and you're feeding back some of that telemetry as context and MCP servers. How do you think about delivering the right amount of context from those servers? Yanbing: So we definitely think, you know, from when we were delivering our MCP service, it cannot be just a API wrapper itself. So there are certain things that we do need to become much more specific. Formatting matters. What type of error messages matters. How we give agent a smart way of query as opposed to just, you know, give it data. I think the more interesting thing I'm also seeing, do customers build with MCP? Certainly MCP provides this universal way of accessing lots of information from different type of AI agents or software tool. In observability, I also see this interesting battle or comparisons of where AI gets build. Observability is definitely going beyond just observe, meaning generate telemetry. Even without AI, observability is no longer just, generate massive amount of data. It's already about how we correlate the data, how I surface the right insight to the user. But now with AI, how you take that into much more intelligent investigation and decision and remediation and action taking. You know, if I were a Datadog customer, I'm constantly thinking about, where do I consume that Datadog data? Do I take it out as MCP, whether I'm doing a root causing, where I'm doing development, likely I will need a lighter set of data. You know, do I access it that way? Or do I access the more curated or opinionated, however you call it, the AI agent that we've built for them, where they go to another vendor that might providing AI agent by synthesizing the world's data from everywhere. I think it's becoming an interesting debate of, what is the right approach? Ryan: Do you think there's still a place for that team dashboard? And are you thinking of, you know, new ways to consume the telemetry and results? Yanbing: I would expect, maybe the access to the beautiful dashboard that we all love, you know, in the Datadog UX. That will remain. Because at certain times, it's just a picture is worth a thousand words. But there is increasingly this new surface of, I want to have Datadog everywhere I go. You know, maybe I want to access that through Slack. And I can even generate or view a particular dashboard in my Slack, or in my CLI, or in my IDU, or wherever... I would say, wherever the work is happening. So we definitely see Datadog were observed in general goes to wherever the work is happening. And what I think is, that doesn't necessarily replace the need for a set of, you know, consistent, curated, well-defined dashboard, because those are just the things you know. I just talked to a AI principal engineer from one of our customers and they were saying, that still, sometimes, even though we have a natural language based interface to tell me what's happening, what's my choices? What's wrong with my logs? Give me this dashboard. It's simply too slow, versus, they know exactly the few clicks to get the information they want. So obviously, AI is extremely good at processing huge amount of information that's beyond human ability to process in a short period of time. For example, in a complex incident investigation, you have all the type of dashboards and telemetries at your fingertips. But still today's modality is, you get into a war room with tens of people, or even, very complex organizations, hundreds of people. And it's interesting, most people's objectives in those situations is not to find the root cause. It's actually to prove innocence, so that they are not part of the problem. And with AI, you know, all of that human-heavy process and time is short circuited by, you know, AI agent immediately understand the context. Have all your histories. Memories about what has happened in the past. Understand, you know, can see all these telemetry, can understand what has changed in the system, and can investigate in parallel, in almost infinite fashion, and get to that root cause before a human can open up their computer. Because, you know, when we open up our computer, get on a Zoom, that's a few minutes. I was talking to a customer, they were running our SRE investigation agent, and they have about 100 agents in the past few weeks, and they have pretty robust statistics, and their average is three-plus minutes, get to that point. And, without any human. So I think there are certain things it's clearly... And that AI didn't open any dashboards. (Laughs) Ryan: (Laughs) Yanbing: So it's gotten to a... It would be attaching a bunch of reasoning and evidence and dashboard to explain to the human. But the AI was able to do that in minutes without going through that. So. To answer your question, I think dashboards remains as useful, because sometimes they're still just, you know, the most direct way to understand and consume information a very reliable way. But customers are using all kinds of different services to access the same information. And AI agent doesn't necessarily read dashboard, they go to the source. And also dashboard can be done just in time. Lots of our customers are asking our AI agent to say, you know, I'm seeing some of this, but I want to compose a new dashboard that show me this particular view that's pertaining to the current investigation I'm trying to do. So it's not going away, but it's how we use it. Where we use it is probably changing. But we're still obsessed with, you know, details of data, what we call data viz within our organizations. How we're graphing things, dashboarding things. You know, displacing things. This is definitely details that we still obsess over. Because I don't think humans are going away from operations. You know, one of my favorite story to tell is, you know, today we have commercial trucks running at 70 miles per hour without a human. I don't expect, you know, software system, whether, the part where building it, where the part where running it doesn't have human involvement anytime soon, because it's just far more complex in terms of the radius, the dependencies, and the type of actions you can take in a complex system. Ryan: Yeah. Yanbing: I think, to me, deeply understanding agent behavior, is it working as you intended? It's really just the first step. Because just like, how we build any services, you know, making sure it works, that's the first step. But then soon you get to the point that, can I secure it? Can I trust it? And I think that's also becoming an interesting problem. Because I don't think... How we secure AI is not yet a fully resolved. We are probably a few steps ahead on agent observability than, you know, what is the right security approach? And AI security is also a multi-edged sword. Because AI can be your target. AI can be your attacker. AI can be the vector that you deliver attacks, or... So it's a very multifaceted... I think that's also something we see has close affinity to observability, because security used to be viewed as a different silo. But with AI, what observability demands and what security demands is actually overlapping quite a bit. It's basically, you understand each and every single action your agent is taking every step. And that's what we're already doing in observability. But how do you use that also to make sure we solve the security use case? I think that's an interesting area that we're also very interested in. And then there's the third question I think we haven't also solved, is the tokenomics, or tokenmaxxing, or the ROI question. Because when we were building large distributed cloud system, you know, kind of your footprint grow with your business. You have more users, you have more cloud, and you observe more. So I think the scaling factor is a lot more directly affiliated with some levels of business growth, where the value out of AI is completely different. So I think this is also an interesting new frontier. It's probably started with observability of understanding tokens and consumption through code efficiency at the primitive level, but how do you link that to also, business outcome? Because here is cost. But AI, we all know, is some... Whatever business value we deliver. So, I think, you know, those are just I see the more advanced topic, but potentially built on the same foundations of, you need to deeply understand every single actions and decisions and steps that your AI agent is taking. But, yeah, just want to make that comment. You know, I don't think we've focused on some of these other topics, but I think they are important, they're related, and they're still not solved. Ryan: Like, there could be these interesting combination use cases, right? Like even like model routing for certain tasks. You could look at the costs, look at the, eval the performance, and then automatically pick models for that. Yanbing: Yeah, we definitely are already doing that for ourselves. Yeah, we... Both static as well as dynamic choice of models, and we both use off the shelf models. We also build our own models. We use large models, small models, and still there are certain things, you give it to a basic ML model. That's still the most reliable way. So, yeah. I think all of those things, it's not just, you know, observing, but also, you know, think about how you truly get the right value out of all the AI investment. (MUSIC PLAYS) Ryan: Well, it is that time of the show where we shout out somebody who came onto Stack Overflow, dropped some knowledge, shared some curiosity, and earned themselves a badge. Today we're shouting out the winner of a Great Answer badge. Somebody who dropped an answer that got 100 or more points. So congrats to "insert username here" for answering, how can I get a character array from a string? If you're curious about that, we'll have the answer for you in the show notes. I'm Ryan Donovan, I edit the blog, host the podcast here at Stack Overflow. If you have questions