loading
loading
AGI Inc. is a research lab building the world's best on-device models that allow AI to autonomously interact with millions of third-party mobile applications. Connect with Div on LinkedIn and check out AGI on X. Congrats to Great Question badge winner Nathan Fellman for winning the badge on their question How does reverse debugging work?. TRANSCRIPT (STACK OVERFLOW INTRO MUSIC) Ryan Donovan: Hello, everyone, and welcome to the Stack Overflow podcast, a place to talk all things software and technology. I am your host, Ryan Donovan, and today we are talking about what it takes to engineer agentic processes, tool use, on the mobile devices. And my guest for that is Div Garg, who is the founder of AGI Inc. So, welcome to the show. Div Garg: Hi. Great to be here. Ryan: Before we get into the topic of the day, tell us a little bit about how you got into software and technology. Div: Yeah. I've always been fascinated by software. You know, I was like, nine years old. We used to have like a bunch of PCs in my house. I was always tinkering around, see what you can build. And... Yeah, I think I was always fascinated, like, what is this box? And, like, how can it do all these things? And this was in India. And then I did a lot of stuff where they're like, it's a bunch of international Olympiads and so on, and they're coming to the US. And this was when I was 16. So this was when I came to the US. I was doing a lot of stuff in physics at the time. So a lot of, like, International Physics Olympiad. And... I just thought I'll become a theoretical physicist. And I was actually thinking like, okay, I should try to get the Nobel medal in physics or something. (Laughs) Ryan: Yeah. Div: And then I think it kind of hit me at that point because, like, one of the biggest things that's going to happen in my lifetime is going to be AI. And I said, as soon as, okay, you have artificial intelligence, and like, all these things are working. That is the next evolution of computers and where things are going. And it'll be very fascinating to work on that. So that's actually what actually got me into computer science, in my undergrad. I was like, okay. This is happening and I should go and study it. And that's what made me into a CS major. And eventually I ended up joining Stanford in the PhD program, focusing on AI and agents. So that was really fascinating. Ryan: Nice. Obviously with the AI, and it's branched out into agents. And I think ever since the Open Claude, everybody's been thinking about how these agents can kind of run their computers. Div: Yeah. Ryan: But today we're talking about running a mobile device, sort of, all on the mobile device. Is that right? Is that what you're doing? Div: Yeah. Our goal is, like, imagine you could replace your whole computer with a phone. And theoretically that's possible because, like, all the computers are already there. Your phones now have almost infinite memory, infinite, like, disk space. And so the question is just like, okay. How do you make your phone useful enough that all your work can be done from your phone? And can you have like a smart enough AI to do this? Ryan: Mhm. Well, and I think that's the thing. You know, anytime I talk about devices and program devices, there's a lot of constraints for it, right? Especially when it comes to AI, where even on desktop, you normally have a hop to the cloud. Div: Mhm. Ryan: How are you thinking about getting the AI to live entirely on the device? Div: Yeah, I think that's where we're doing a lot of innovative work as a company. So I'd say fundamentally, we are a research lab. And a lot of our work is around on device models. So we actually have the world's best models that can run on a device to take actions for you. And they can work across any third-party apps. You can ask it, like, can you call me an Uber? Can you order me something on Amazon? Get you food on DoorDash. It will just generalize to anything. So, like, there's millions of apps out there, and our AI just works across everything. Ryan: Mhm. I mean, world's best device app is a big claim. Big claim. So, are you doing that by creating a specialized model that's just doing tool-calling actions, that sort of thing? Div: No, it's more than that. So it's like, it can actually mimic a human's interaction. Ryan: Mm. Div: So it's like, the industry term for this is computer use or mobile use. So the AI has learned how to interact with the screen. So like, touch, type, scroll. And whatever you can do on the phone, it can basically do the same thing as a human. Ryan: Mhm. What sort of model is that? Is that the sort of transformer model that's gotten all the press? Or is it a sort of more standard deep learning model? Div: Yeah, it's like, a transformer-based architecture. And we do a lot of this stuff ourselves. So what happens is, in most of the industry, people are running on Nvidia hardware. If you're running on all these, like, V200, V300, in the cloud, including Anthropic. And then you can compile models to CUDA, which is the standard framework for running code on these GPUs. But when you're working with mobile devices, you're actually not able to use Nvidia. Because all this is powered by edge computing chips. And a lot of that is, like, Qualcomm or other providers. And so we have to actually write all the kernel-level code ourselves. And so we have done a lot of innovation. Like, how do you make the the compiled code for the model, on the mobile processor, very fast and instantaneous? How do you make sure it's optimized for your memory? Ryan: Mm. Yeah, we've done some work with articles with Qualcomm. And they... Talking a lot about quantization. Div: Yeah. Ryan: Also we were talking about CUDA, you're talking about the programming on the GPU. Div: Mhm. Ryan: Is there a sort of, like, mix of those you're using? Are you both quantizing and programming directly on whatever Qualcomm chips there are? Div: Yeah, I think we do both. So, like, quantization is a big part of making the model smaller. And then you have to do a lot of compilation. We actually just wrote this technical blog post that got a lot of views. We have an auto compilation engine. So we basically taught an AI agent how to write the best optimized code for a particular mobile chip. And can figure it out itself. And it can, like, self-optimize. Ryan: Oh, wow. So what's the code that you use to train that? Like, obviously you have to train it on the best optimized mobile code. Are you writing that yourselves? Do you... Does Qualcomm say, here's, like, the best we have? Div: Yeah. Yeah. We're writing a lot of it ourselves. So like, when you're working on Qualcomm libraries. The thing is, like... A lot of that is not really public. And there's not enough, like, standards or community around it. So a lot of that is like, something you have to figure out yourself. They have a lot of private APIs. Ryan: Yeah. Div: Luckily we have a collaboration directly with Qualcomm. So we get some sort of like, touchpoints. But we had to figure out a lot of, like, the [inaudible] case ourselves, on how to make this really, really good. Ryan: Yeah. I mean, that's, that does seem like the rub. So... For the optimizations there, what is the sort of... I mean, if you're teaching it to write highly optimized code on a mobile device for any given code, any given, sort of, software problem, what are the sort of optimizations that you teach it? Or is it broader than that? Yeah. Div: Yeah. I would say it's broader than that. One thing we try to do is like, suppose we say our agent wants to call... Like, do something for you. So we want you to, say, even do something like, book me an Uber to my office. And this is something that you could do yourself. But if the agent is doing for you, you want it to be faster than, like, the human. And so a lot of them goes, how do you beat people on speed? And so, can you make this, like, really, really fast? And that because the model has to be small enough, the code has to be efficient enough, to make sure we can get that speed performance. So that's one thing we've been doing. Okay, how do we get the best speeds possible? And we're actually number one in the industry also when it comes to reliability. So our agents actually reached an accuracy of 97.4% on one of the most recognized leaderboards that's called Android World. This is what everyone in the space is using for Android-based agents. And I think it's just been a big validation of our approach. Like, we couldn't have done it without writing all the kernel code ourselves. Ryan: Yeah. And how do you verify that it's both, you know, the most efficient stuff, and it produces accurate results? Div: Yeah. We actually have a bunch of testing suites we have created internally. We actually built the first... (Laughs) Sandbox environment for these kind of agents, way back in 2024. Ryan: Yeah. Div: New York Times actually wrote an article about us. So this is a public app where, it's called realevals.xyz. And we cloned the top ten websites on the internet. So we made a clone of, like, DoorDash, LinkedIn, Amazon. And these are, like, actual... It looks like a real website. We actually had a clone of United, and then United gave us a legal notice to take it down. Ryan: (Laughs) Div: Because it was just too alike. (Laughs) Ryan: Yeah, yeah. (Laughs) And then you test it on... When you say clone, what do you mean by clone? Div: Yeah, we actually just, like, literally built a... A facsimile of the actual website. Obviously you can't book a real flight on it, but it's very good for, like, training an agent. (Laughs) Ryan: Yeah, yeah. When you're, you have these sort of facsimiles and training on it, what did you find that your models and your agents struggled on? What was the thing that tripped them up? Div: I think there's obviously a lot of things. One big thing we found was just like, how do you build safety? And align with humans? So for example, like, say... Again coming back to flights. So if I say, like, go and book me a flight to... Maybe like, Japan or something. Like that might be a thousand-dollar flight, you don't want it to automatically go choose something and book it for you and just tell you, like, hey, by the way, I got this flight. Ryan: Yeah. Div: And like, here you go. You want it to be more like human in the loop. Ryan: Yeah. Div: So how do we show choices to the person? Okay? Like, these are the top five. Which one do you want? How do we show confirmation to the user? Okay. Like, how do you get the agent to stop at the checkout and say, like, okay, this is what it's going to cost you. And, do you want to proceed, yes or no? And so, I think that's been one of the learnings. Like, how do you build this kind of like, human interaction? Ryan: Right. Div: Where it takes care of all the boring parts that you don't want to do, but you still have the control and the autonomy. Ryan: Yeah. I definitely feel that. Because, you know, even with a regular search and finding the one I want, I'm like, I don't want to pull the trigger on this. I...What if I did something wrong? Div: Yep. Ryan: I could imagine the agentic commerce part, of actually pulling the trigger and spending the money, is kind of the hardest piece of the safety puzzle. Div: Yes. Ryan: So, did you find that there's any way to automate that, to sort of build the safety and the trust into the process? Div: Yeah. Like, a lot of it is part of the training process. So we do, like, we take moderation very seriously, for example. So, like, our AI again, can use any app on the phone. So there's millions of apps. We try to steer a lot of that towards the... Kind of like, the safer side. So we prohibit the AI to work on banking apps, for example. Because we don't want people to use it for financial transactions autonomously. Ryan: Sure. Yeah. Div: Yeah. And then we are building kind of safety mechanisms as part of the training. So, say, it can refuse tasks, or it can say, like, this seems to be risky or malicious or whatever. And then we pass over the control back to the human. So, something... That's like, something that you sort of figure out. And I think that that's what it takes to build a real-world system, that people are actually using. Ryan: Mhm. So, were you able to do any sort of like... You know, the android, you can run it in a sort of emulator, right? Div: Yep. Ryan: Did you find... First of all, did you do that, I assume? And then, did you find that that gave accurate and real results? Div: I would say it was a good start for us. We actually found it does not generalize that well. Because what happens is, emulators are limited. Like, number of apps you can run. What you can do. I think that has been a good start for us. We have been doing that. We were doing that, actually, earlier this year. And then we actually transitioned to using real phones. So a lot of the agents now are running the company are running on actual phones that we have set up. We actually have devices where the agent can improve itself, become better over time. Ryan: Yeah. I mean, I remember a friend of mine was building mobile games. He would go to every mobile device conference just to get the devices. To test on. Are you able to run this on all hardware, or is it just the Qualcomm-based stuff? Div: Yeah. This can go on all hardware. I mean, just, Qualcomm is one of our partners. We've just done more specific optimizations for them. But this can be running on everything else too. Like, AMD, Nvidia, Intel. Ryan: Was it more difficult to optimize on those, that you didn't have the sort of direct access to a partner? Div: Yeah. That depends. Yeah. It's like... It obviously helps if you have a reference device. So, for example, like, we work with a lot of OEMs. And those OEMs are using Qualcomm chips. So it's very easy for us to go, like, okay, here's the reference device we're launching on, or optimizing the agent on. And we're starting to do that with other partners, too. So, like, for example, if we are doing something with tablets or laptops, a lot of them might be like, AMD or Intel. As a processor. So now we're starting to, like, work with those companies and also the chip manufacturers. But I think having a direct relationship definitely helps a lot. Because there's a lot of internals that are not public. Ryan: Yeah. And there's a... You know, anytime I've seen anybody doing, like, on-chip programming, whether it's SIMD or the kind of general purpose GPU programming, it's a very specific world of programming, right? When you were sort of doing that and optimizing that, did you find that, you know, you didn't have enough training data for that? Div: Yeah. Data is always a big thing. I think you always want more of it. Ryan: Yeah, yeah. Div: We... I think we collect a lot of data ourselves, because it's just like, there's not enough data out there. Especially on, like, how to use these interfaces. So we actually talk to a lot of... We talked to 20 different vendors. And all of them said, like, okay, yeah, we, we have not done this before, because it's just so new. Ryan: Right. Div: And so we decided, like, let's go and do this ourselves. So that's, like, something we have been doing. And so that's been great. We're also trying to build a lot of synthetic data now. So we can have emulators, automatically, like, an agent that can run and generate these data sets. And we can train on that on top. Ryan: I've heard a lot of people using synthetic data effectively. But I've also heard of, you know, the risks of things like model collapse, and it just sort of like... Pushing towards a mean. Div: Yeah. I think it depends how you generate it. So we have been very careful in generating actually hard scenarios. So what happens, like, you can generate a lot of normal data, but there are like, hard scenarios which are just hard to replicate. It's like, it's very, extremely difficult to like, force a phone into a situation where it might do something glitchy, or like, maybe the... You want to simulate bad behavior, right? Ryan: You want to get the edge cases, right? Div: Yes, yes, yes. And so that's where we have been like, sort of spending a lot of time with simulations, on how do you get to the edge cases? That are very hard to create in the real world. Or, like, manufacture. Ryan: So how do you simulate edge cases? Like, if it's hard to actually create in the real world, does it become easier in a simulation? Div: Yeah. I can give you an example. So one thing we can do is, like, some sort of poisoning attack. So for example, like suppose you're on an email app, and suppose someone hides an invisible instruction to the agent, like, okay, send me a... Buy me this money on this crypto link or something. And we can simulate a lot of those kind of scenarios. Where, like... And then that allows us to make the agent more safe. Ryan: So, yeah. That brings up a lot of... Obviously there's a lot of security risks, and you're talking about, you know, forbidding certain apps. Div: Yeah. Ryan: Does that mean you have to have a sort of blacklist of apps that you can't use? Div: I think we do have that for some things. We try to keep it small. It's mostly about, we try to make it like, okay, can the agent itself learn to refuse some things? Ryan: Mm. Div: So it's like, how can we make that kind of like a blacklist, as part of the intelligence itself? So we don't have to like, hard code everything. Ryan: So if it's... If it looks like a banking app, it smells like a banking app, it's gotta be a banking app. Div: Yeah. And I think Google also publishes, it's a banking app. So like, you can't really have a banking app that that says, like, it's not a banking app. (Laughs) Ryan: Yeah. Well, I mean, there are banking apps that have gotten in trouble. Div: Yeah. Ryan: Yeah. For not really being banking apps. (Laughs) So that definitely seems like a hard one, if, like, you know, even the law and the people don't quite agree on what is a banking app, right? Div: Yes, yes. Ryan: Have you done any sort of testing on that? And have you been sort of surprised on what it has chosen or forbidden? Div: Mm. So far it's been good. I think, like, we've seen some interesting interactions around dating apps, for example. Ryan: Oh, okay. Div: So someone might just try to use it on, like, uh, I don't know, like a Tinder, for like, swiping on profiles. And it's kind of like... Do you want to allow something like that or not? And it's like, it's kind of a gray zone. (Laughs) Ryan: Yeah. That seems... (Laughs) Like it's going to make dating way worse. Just automating swiping right and left, right? Div: Yeah, yeah. So we try to like, we want to incentivize users from using the AI like that. Ryan: Yeah. Div: But... Yeah, so this is definitely edge cases, like these kind of gray scenarios. Ryan: Yeah. Well, I mean there's a line between disincentivizing and forbidding, right? Div: Yeah. Like, obviously it's obvious. Like, the problem is there's always boundaries. So it's like, maybe... If there's like something like, suppose you're using it for curiosity. So we want to allow you to just, like, this is possible. But then you're a heavy user that just like, started to use this for spamming people. And that's bad. And then we want to ban you, in a sense. Ryan: Yeah. Div: Or like, take some sort of action. But it kind of becomes, like... Initially you want to keep it as a playground where people are like, exploring and figuring out what's possible. And as it becomes more mature, then we like, put more filters, and like, more guardrails. But initially we want to be kind of like, yeah, let's go and try to use it... It's kind of like, having this vision where, like, okay, everything is voice-enabled. So whatever I'm doing with my hands, with the keyboard, with like, touching and typing. Now suppose I can just use the phone with voice. And this was a vision like Siri, Apple has had with Siri. So we want to really go and show people that's possible. Like, I can just go and do anything handsfree. Ryan: Yeah. Div: And then... Eventually, like, put more guardrails and make it, like, safe. Ryan: Yeah. But right now it's kind of in a YOLO mode. And like you said, a research lab. Div: Yeah. Initially. It's also, like, a beta product. So I think we are still, like, growing the users. We actually work with a lot of OEMs. So... Like, I can't name them right now because of like, some partnership agreements. But we work with some really big mobile companies. And they understand their users. Ryan: Mhm. Div: They also have their own guardrails. So it's very easy to, like, make sure we do the right things. Ryan: Yeah. I wonder if you have any... If you're thinking about guardrails in terms of, like, having the AI agents doing non-huma behaviors. Like going into a dating app and, uh, harvesting all the pictures and profiles of everybody on there, or something like that. Div: Yeah. I mean, I think it's not something we want to allow, obviously. Ryan: Sure. Yeah. Div: It's also, I think we take screenshots very, like... I think we really care about screenshots that are being created. So, for example, like, even we don't have access to any screenshots. And we don't save screenshots. So unless someone says, like, explicitly, have the AI go take a screenshot on the Tinder app and like, save it... Like, this is something you could actually do yourself. So in that case, it's like, we're not creating a new door. And then there's ways we can catch if someone is trying to create a bot out of that or something, we can do that. Ryan: Yeah, yeah. I mean, if somebody said, like, take a screenshot of every profile on there. Like... And then swipe left, like... I don't know. If we've learned anything about AI and AI agents, if it is possible, it'll eventually get done. Right? Div: Yeah. Yeah, I think it's like... (Laughs) People want all sort of things. Ryan: Yeah. Div: And I think the argument is just like, it's a tool. And it's like, the actors, and how people use it. Ryan: Yeah. So when you were... You're sort of, you know, testing and building, and... What is the thing you're sort of struggling with now? What's the single biggest challenge with it now? Div: Hm. It's just like, we see users are fascinated with the product. So they try it for the first time, like, whoa, I didn't realize I can do all these things. And I can just talk to my phone. And we're just trying to figure out, like, what are the habit-ification things we can do? So, like, how do we get people started using it all the time? And... I think we see like a lot of things. Like, we actually have users who are using it to play video games, for example. And I was like, whoa. Like, we'd have never expected someone to do that. Ryan: Yeah. Div: And it's just, it's just very interesting to see all the different ways people are using it. And then we're like, I think people are starting to also build skills on top. So we have this like way where you can like give it, like, this is like how I like to do things. I have this food allergy. I'm allergic to peanuts. So when I'm ordering food, like, take that into account. So you can have the different skills, like how you build with Claude. So you can have something done into the end of the product. Ryan: Mhm. Div: And that's really fascinating. So we have people, like, starting to build routines. They're codifying a lot of the ways they spend time. So we actually have this, like... This is something I use a lot. It's like, I have this skill which calls me an Uber to my office every morning. Ryan: Uh huh. Div: And I just automatically called an Uber. And then it's... It's really cool, because it's like, I don't have to do anything. Ryan: Yeah. Div: And so you can start building this, like, really cool automations on top. Ryan: Yeah, yeah. Do you find that the smaller on-device model has the sort of accuracy? Does it run into any problems at all? Div: So it's really good. We do supporting of Claude mode for hybrid reasoning. Ryan: Mhm. Div: So it depends on, like, what you want. Like, a lot of tasks on the phone are actually not very long, in a sense. So, it might be something like, okay, order me coffee on Starbucks. And you can do things like that in like, 20 to 30 steps. So you don't really need really complex reasoning or capabilities. But then if I want to do something that might take 100-plus steps, then I think we have a Claude mode that allows you to do more things. And we are starting to do stuff where it can also intelligently switch between different modes if need be. So you can have a smaller model, then a bigger model. And depending on the complexity of the task, it can decide which to use. Ryan: Mm. So, I'm not sure that everyone listening understands, like, the number of steps that are involved in generalized tasks. Can you give us an example of what a 20-step task is versus a hundred-step task? Div: Sure, sure, sure. So, for example, if I say, like, book me an Uber, that's like, maybe five steps. It's like, okay, the AI has to go and open the Uber app. We search for that destination, choose, like, Uber Rides or something different. Click confirm. So it's like, it's five steps. So it's like, pretty small. You might have something like, okay, now suppose I want to order something like a coffee or Starbucks. And there might be a lot of customization. So a lot of times you spend on the menu. So like, you have to open the menu for the Starbucks app. Then you have to think about customizations. It has to maybe like, select all the different, like, things you want, flavors and stuff. And so, like, you spend a lot of budget on just those customizations. And that can get you to, like, up to 20 steps by the time you check out and you actually order the coffee. Ryan: Yeah. Div: And also, that's actually a pretty complex use case, for a personal assistant on a phone. And, I'll say something like, that takes a hundred steps is probably like, more of an enterprise use case. So it might be something like, okay, I want you to go and organize my spreadsheets. Or I want you to go and answer all my emails from the last day. And then those things can take time because it's running in a loop. Ryan: Right. Div: But like, for any single task, it's actually pretty, pretty fast. And like, you don't need to like, cross too many steps. Ryan: Mm. The coffee being more complex than an Uber... I think you've validated every hack comic out there complains about coffee. Div: (Laughs) Ryan: When you think about what would take this out of beta, what in your mind would be the difference between the beta and the finished product? Div: Yeah. Well, it's just, I think a lot of this is like, making it scalable to different languages and different regions. We actually have a lot of users in Asia. So we actually went viral in Thailand for some reason. We don't even know why. I think a lot of people in Thailand started using us overnight. So we got, like, around like 20,000 users popping up, and we're like, whoa! Ryan: Yeah. Div: How are they finding out about us? And they're using us in, like, Thai... In local apps. So they have their own version of Uber. They have their own shopping apps. And the AI actually works everywhere. And like, they're talking to it in Thai, it's talking back to them in Thai. And it's like, whoa, like, we did not really expect it to be like that here. Ryan: Yeah. Div: Because that's literally out of our testing. We have not done any benchmarking or testing on those kind of scenarios. Ryan: Yeah. Div: So a lot of productionization is like, how do you make it multilingual? How do we support more niche applications in different geographies? And make sure everyone is having a good experience as we, like, make this public. Ryan: Yeah. So, I guess you'd never tested it in Thai or other languages, right? Div: Yeah. It's mostly been on English. I think we started to branch out to some new languages, but our agentic models are inherently multimodal. It's just been surprising how well they work. Ryan: Yeah. I wonder if you... Did you find other countries have different use cases? You know, I've heard that Africa, for example, basically, we skipped the desktop and went right to mobile. Div: Yeah, yeah, I think that's very true. The same thing for India, for example. So India is like, very much a mobile-centric world. And a lot of people in the world actually don't use, don't even have desktops. So I've seen that, even in Europe, for example, people might have a tablet at the maximum, but everyone's just on a phone otherwise. And I think that's been great. I think, definitely, as you said, like, people have very different use cases in different places, in different geographies. The shopping has been a big one. That's universal, but people like to use different apps in different languages. And in particular, do they... What they like to do. Food has been a good one too. People are always like, order me some ice cream on the food app they like to use again, or pizza, stuff like that. So it's mostly convenience. Like, entertainment also has been a big one. Okay? Like, play me something on Spotify, find me something on YouTube. It's very interesting to see all those things. Ryan: Yeah. Div: We're very excited about, like, how can we make these kind of handsfree actions sort of like, universal? Ryan: Mhm. Div: So, one very fun development we recently had is we got the founder of Android, Rich Miner, to join as an official advisor. And he believes what we are building can be the next operating system after Android. And... We're just trying to build this world where, imagine, like, you can do anything handsfree. Like, you might not even have to use a phone anymore. Like, you have some sort of like... Smart glasses or smart pendants, and you have AI running there, and you just talk to it. It's like, okay, I want to like go get all these
AGI Inc. is a research lab building the world's best on-device models that allow AI to autonomously interact with millions of third-party mobile applications. Connect with Div on LinkedIn and check out AGI on X. Congrats to Great Question badge winner Nathan Fellman for winning the badge on their question How does reverse debugging work?. TRANSCRIPT (STACK OVERFLOW INTRO MUSIC) Ryan Donovan: Hello, everyone, and welcome to the Stack Overflow podcast, a place to talk all things software and technology. I am your host, Ryan Donovan, and today we are talking about what it takes to engineer agentic processes, tool use, on the mobile devices. And my guest for that is Div Garg, who is the founder of AGI Inc. So, welcome to the show. Div Garg: Hi. Great to be here. Ryan: Before we get into the topic of the day, tell us a little bit about how you got into software and technology. Div: Yeah. I've always been fascinated by software. You know, I was like, nine years old. We used to have like a bunch of PCs in my house. I was always tinkering around, see what you can build. And... Yeah, I think I was always fascinated, like, what is this box? And, like, how can it do all these things? And this was in India. And then I did a lot of stuff where they're like, it's a bunch of international Olympiads and so on, and they're coming to the US. And this was when I was 16. So this was when I came to the US. I was doing a lot of stuff in physics at the time. So a lot of, like, International Physics Olympiad. And... I just thought I'll become a theoretical physicist. And I was actually thinking like, okay, I should try to get the Nobel medal in physics or something. (Laughs) Ryan: Yeah. Div: And then I think it kind of hit me at that point because, like, one of the biggest things that's going to happen in my lifetime is going to be AI. And I said, as soon as, okay, you have artificial intelligence, and like, all these things are working. That is the next evolution of computers and where things are going. And it'll be very fascinating to work on that. So that's actually what actually got me into computer science, in my undergrad. I was like, okay. This is happening and I should go and study it. And that's what made me into a CS major. And eventually I ended up joining Stanford in the PhD program, focusing on AI and agents. So that was really fascinating. Ryan: Nice. Obviously with the AI, and it's branched out into agents. And I think ever since the Open Claude, everybody's been thinking about how these agents can kind of run their computers. Div: Yeah. Ryan: But today we're talking about running a mobile device, sort of, all on the mobile device. Is that right? Is that what you're doing? Div: Yeah. Our goal is, like, imagine you could replace your whole computer with a phone. And theoretically that's possible because, like, all the computers are already there. Your phones now have almost infinite memory, infinite, like, disk space. And so the question is just like, okay. How do you make your phone useful enough that all your work can be done from your phone? And can you have like a smart enough AI to do this? Ryan: Mhm. Well, and I think that's the thing. You know, anytime I talk about devices and program devices, there's a lot of constraints for it, right? Especially when it comes to AI, where even on desktop, you normally have a hop to the cloud. Div: Mhm. Ryan: How are you thinking about getting the AI to live entirely on the device? Div: Yeah, I think that's where we're doing a lot of innovative work as a company. So I'd say fundamentally, we are a research lab. And a lot of our work is around on device models. So we actually have the world's best models that can run on a device to take actions for you. And they can work across any third-party apps. You can ask it, like, can you call me an Uber? Can you order me something on Amazon? Get you food on DoorDash. It will just generalize to anything. So, like, there's millions of apps out there, and our AI just works across everything. Ryan: Mhm. I mean, world's best device app is a big claim. Big claim. So, are you doing that by creating a specialized model that's just doing tool-calling actions, that sort of thing? Div: No, it's more than that. So it's like, it can actually mimic a human's interaction. Ryan: Mm. Div: So it's like, the industry term for this is computer use or mobile use. So the AI has learned how to interact with the screen. So like, touch, type, scroll. And whatever you can do on the phone, it can basically do the same thing as a human. Ryan: Mhm. What sort of model is that? Is that the sort of transformer model that's gotten all the press? Or is it a sort of more standard deep learning model? Div: Yeah, it's like, a transformer-based architecture. And we do a lot of this stuff ourselves. So what happens is, in most of the industry, people are running on Nvidia hardware. If you're running on all these, like, V200, V300, in the cloud, including Anthropic. And then you can compile models to CUDA, which is the standard framework for running code on these GPUs. But when you're working with mobile devices, you're actually not able to use Nvidia. Because all this is powered by edge computing chips. And a lot of that is, like, Qualcomm or other providers. And so we have to actually write all the kernel-level code ourselves. And so we have done a lot of innovation. Like, how do you make the the compiled code for the model, on the mobile processor, very fast and instantaneous? How do you make sure it's optimized for your memory? Ryan: Mm. Yeah, we've done some work with articles with Qualcomm. And they... Talking a lot about quantization. Div: Yeah. Ryan: Also we were talking about CUDA, you're talking about the programming on the GPU. Div: Mhm. Ryan: Is there a sort of, like, mix of those you're using? Are you both quantizing and programming directly on whatever Qualcomm chips there are? Div: Yeah, I think we do both. So, like, quantization is a big part of making the model smaller. And then you have to do a lot of compilation. We actually just wrote this technical blog post that got a lot of views. We have an auto compilation engine. So we basically taught an AI agent how to write the best optimized code for a particular mobile chip. And can figure it out itself. And it can, like, self-optimize. Ryan: Oh, wow. So what's the code that you use to train that? Like, obviously you have to train it on the best optimized mobile code. Are you writing that yourselves? Do you... Does Qualcomm say, here's, like, the best we have? Div: Yeah. Yeah. We're writing a lot of it ourselves. So like, when you're working on Qualcomm libraries. The thing is, like... A lot of that is not really public. And there's not enough, like, standards or community around it. So a lot of that is like, something you have to figure out yourself. They have a lot of private APIs. Ryan: Yeah. Div: Luckily we have a collaboration directly with Qualcomm. So we get some sort of like, touchpoints. But we had to figure out a lot of, like, the [inaudible] case ourselves, on how to make this really, really good. Ryan: Yeah. I mean, that's, that does seem like the rub. So... For the optimizations there, what is the sort of... I mean, if you're teaching it to write highly optimized code on a mobile device for any given code, any given, sort of, software problem, what are the sort of optimizations that you teach it? Or is it broader than that? Yeah. Div: Yeah. I would say it's broader than that. One thing we try to do is like, suppose we say our agent wants to call... Like, do something for you. So we want you to, say, even do something like, book me an Uber to my office. And this is something that you could do yourself. But if the agent is doing for you, you want it to be faster than, like, the human. And so a lot of them goes, how do you beat people on speed? And so, can you make this, like, really, really fast? And that because the model has to be small enough, the code has to be efficient enough, to make sure we can get that speed performance. So that's one thing we've been doing. Okay, how do we get the best speeds possible? And we're actually number one in the industry also when it comes to reliability. So our agents actually reached an accuracy of 97.4% on one of the most recognized leaderboards that's called Android World. This is what everyone in the space is using for Android-based agents. And I think it's just been a big validation of our approach. Like, we couldn't have done it without writing all the kernel code ourselves. Ryan: Yeah. And how do you verify that it's both, you know, the most efficient stuff, and it produces accurate results? Div: Yeah. We actually have a bunch of testing suites we have created internally. We actually built the first... (Laughs) Sandbox environment for these kind of agents, way back in 2024. Ryan: Yeah. Div: New York Times actually wrote an article about us. So this is a public app where, it's called realevals.xyz. And we cloned the top ten websites on the internet. So we made a clone of, like, DoorDash, LinkedIn, Amazon. And these are, like, actual... It looks like a real website. We actually had a clone of United, and then United gave us a legal notice to take it down. Ryan: (Laughs) Div: Because it was just too alike. (Laughs) Ryan: Yeah, yeah. (Laughs) And then you test it on... When you say clone, what do you mean by clone? Div: Yeah, we actually just, like, literally built a... A facsimile of the actual website. Obviously you can't book a real flight on it, but it's very good for, like, training an agent. (Laughs) Ryan: Yeah, yeah. When you're, you have these sort of facsimiles and training on it, what did you find that your models and your agents struggled on? What was the thing that tripped them up? Div: I think there's obviously a lot of things. One big thing we found was just like, how do you build safety? And align with humans? So for example, like, say... Again coming back to flights. So if I say, like, go and book me a flight to... Maybe like, Japan or something. Like that might be a thousand-dollar flight, you don't want it to automatically go choose something and book it for you and just tell you, like, hey, by the way, I got this flight. Ryan: Yeah. Div: And like, here you go. You want it to be more like human in the loop. Ryan: Yeah. Div: So how do we show choices to the person? Okay? Like, these are the top five. Which one do you want? How do we show confirmation to the user? Okay. Like, how do you get the agent to stop at the checkout and say, like, okay, this is what it's going to cost you. And, do you want to proceed, yes or no? And so, I think that's been one of the learnings. Like, how do you build this kind of like, human interaction? Ryan: Right. Div: Where it takes care of all the boring parts that you don't want to do, but you still have the control and the autonomy. Ryan: Yeah. I definitely feel that. Because, you know, even with a regular search and finding the one I want, I'm like, I don't want to pull the trigger on this. I...What if I did something wrong? Div: Yep. Ryan: I could imagine the agentic commerce part, of actually pulling the trigger and spending the money, is kind of the hardest piece of the safety puzzle. Div: Yes. Ryan: So, did you find that there's any way to automate that, to sort of build the safety and the trust into the process? Div: Yeah. Like, a lot of it is part of the training process. So we do, like, we take moderation very seriously, for example. So, like, our AI again, can use any app on the phone. So there's millions of apps. We try to steer a lot of that towards the... Kind of like, the safer side. So we prohibit the AI to work on banking apps, for example. Because we don't want people to use it for financial transactions autonomously. Ryan: Sure. Yeah. Div: Yeah. And then we are building kind of safety mechanisms as part of the training. So, say, it can refuse tasks, or it can
document fetched 2026-10-24 10:03 · canonical https://stackoverflow.blog/2026/09/29/your-phone-is-ai-s-newest-hardware