Data is the Hard Part

extract

Bharat Patel is currently Accenture’s AI and data lead for its defense portfolio, a former enlisted Navy sailor, and a veteran of the Army acquisition world, with experience at MITRE and across the Department of Defense. We’re here today to talk about why data is the hard part in making AI work to win wars for you — and what standards, integration processes, and governance you need to turn fun tech and cute little language models into things that can actually serve the warfighter. We’re also going to talk about the dark side of data — how poisoning can happen, the enemy’s vote in the datasets you build to find things on the battlefield, and what it’s like to be a civil servant trying to make big programs happen inside the acquisition bureaucracy. Our conversation covers: - Why data is the real bottleneck for military AI, why “AI-ready data” is a myth, and why autonomous tanks are further off than you might think, - Ukraine’s autonomous systems as a case study: how years of collecting, labeling, and learning from battlefield data created the foundation for autonomy, - The boring side of AI: how pipelines, standards, governance, and testing and evaluation determine whether models ever make it into the field, - The adversary’s AI toolbox — including camouflage, deception, electronic warfare, and poisoned data — and how to test against those threats. - The 4D chess game of building programs inside the Pentagon: navigating requirements, funding, acquisition rules, and the people who make or break a program. Listen now on your favorite podcast app. Thank you to Accenture Federal Services for bringing us this episode. The Dark Side of Data Jordan Schneider: Bharat, why is data the hard part? Bharat Patel: We talk about AI-ready data, but in my opinion, data will never be AI-ready. Data will always have its weirdness about it. It’s more about what the AI use case is, because the use case drives the type of data you need and the quality you need it in. We often focus on the wrong problem. Just figuring out which AI use cases you want to go after will help us determine the data quality and how to pursue it. Jordan Schneider: Why don’t you pick a case study and walk us through how data can be used and fit for purpose? Bharat Patel: I’ve had a ton of experience across the Department of War on different AI projects. One of the most famous was Project Maven, started in 2017. The first thing we all struggled with was the data itself. We were trying to build computer vision models against a particular set of targets, and then trying to get enough data that actually had those targets in it. At the start of Project Maven, we were getting a ton of data that didn’t contain relevant targets, so our models weren’t performant once we put them into operations. We quickly learned that we had to continuously collect relevant data to build those models. That was one of my first early inroads into understanding the use case tied to the data, and then getting enough of it to build performant models. Jordan Schneider: How do you get an organization to start taking data seriously, and what is the infrastructure? Let’s start with the cultural piece. How do you spread the gospel on data first? Bharat Patel: First, you have to find your battle buddies who already understand some of the gospel. Once you find them spread across different parts of the organization, you create a culture to bring them up, and they advocate to leadership that this is important. But you can only do that with evidence. One thing I was involved in was through the Army Research Lab. We wanted to build a model for tanks — second-gen FLIR tank data. There was no tank data, so we had to figure out how to get it into an environment where we could apply AI techniques. The team had to build a data collection box, put it right next to the sensor they wanted data from, and collect enough of it to bring back into an environment. Jordan Schneider: What data do you want from a tank? Bharat Patel: At first, it was just the second-gen FLIR sensors. It was more about the sensor technologies, and at that time, it was imagery. But there are other sensors on a tank that we just didn’t get to. If we’d had that data too, now you’re talking about multimodal data and what that can do for autonomy, target tracking, and target execution. We didn’t get that far. We started with baby steps — just imagery — but now you’re seeing an explosion of multimodal data, because that’s the only way you’ll get to autonomy: not through one sensor, but multiple. Jordan Schneider: Let’s ground ourselves with the dream. What do you want all this data to let a tank do in 2035 or so? Bharat Patel: The first thing we’d want a tank to do is make sure our military men and women are safe. More autonomy will save lives. Getting a tank to do things a human can do — we’re still a long way from that, but we can all dream, right? So 2035 gets us an autonomous vehicle that can understand its environment, make decisions, and save human lives as people sit back and watch. Jordan Schneider: Do you still need human beings in tanks? Why isn’t everything a UGV today? Bharat Patel: Because autonomy still goes wrong every once in a while. You don’t necessarily hear about all of it yet, but it should be more evident. We need a human-in-the-lead because you want people to verify what the autonomous vehicles are doing. We also have a manufacturing issue — though those are ramping up due to some things coming out of the White House. Having tons of autonomous platforms out there running wild is just not where we are as a nation. A lot of it goes back to whether we have the right policies in place for these things to autonomously do things. It’s very easy to mistake an enemy combatant for a civilian, or an enemy vehicle for a civilian vehicle. There’s a lot we’re just uncomfortable letting loose as a nation. Jordan Schneider: Waymos can drive better than me, so this seems not impossible. Maybe a Bradley just isn’t built to operate without human beings moving stuff around and pressing buttons. But what you see in Ukraine right now is still a lot of dudes driving trucks, plus a handful of cool things doing casevac or holding a position on a corner of a tree line. It’s been surprising to me that even in a hot conflict — where there are fewer civilians running around and maybe less stress about those questions — it hasn’t gotten further. Is it a data challenge? A compute-at-the-edge challenge? What’s holding us back? Bharat Patel: That’s a phenomenal use case. What you see today in Ukraine is not how it started. For a very long time, there was an active data collection strategy happening, unbeknownst to the people who were there. A lot of the technologists who saw the future of warfare understood they needed to make that data available. They didn’t start with semi-autonomous or autonomous platforms. They started with regular humans flying around with joysticks. But over time, what got recorded — video, different sensors collecting — that data made it back to a central place. People started to realize, “I can build things and make them more autonomous. I can identify targets a little more. I can identify things in this data.” Then they brought it back into the platforms to make them autonomous. I don’t see that happening at mass quantity within the department. There’s no active data collection strategy for machine learning, for continuous machine learning. There are exercises and activities, but many times — because storing data and doing something with it afterwards is resource intensive — it just gets erased, deleted, or dumped into a big storage area where it loses context over time, and people don’t even know it’s there. Jordan Schneider: Your point is that whatever autonomy we’re seeing in Ukraine today is because they’ve spent three years collecting and labeling data. On the one hand, we’re not fighting a war, so there’s less data to collect. But it’s also a lack of focus on the US government’s part to set up the relatively boring things you need — the custody, having all this sitting on a hard drive ready to train on. Bharat Patel: I 100% believe the boring side of AI is now starting to become more popular. The folks who’ve been in this area understand what it takes — data collection, processes, standards, governance, and infrastructure, all the boring stuff. That hasn’t been a high priority. I’m seeing some good things out of the CDAO and their scaffolding initiative that start to get after it. But the department still doesn’t have the right folks across all the spaces to understand how important it is to continuously do the boring stuff of AI to move any of this forward. Jordan Schneider: I guess one of the “silver linings” of fighting a horrific war is that you have a lot of data to train on. To what extent can you get the data you need from training exercises and synthetic data, versus actually needing the real thing to build models that achieve the autonomy you’re hoping for? Bharat Patel: Taking advantage of the exercises the military goes through is one avenue. But there’s always a question of realism — they may or may not be representative of actual warfare. In my mind, the more important thing is the reps and sets: how quickly can we collect data, bring it back into an environment to be trained, do testing and evaluation, and push models back out to where they need to go? It’s really about building a true pipeline that can rapidly move things from one end to the other, versus trying to build reliable AI models based on internal operational exercises. What we saw in Ukraine was a non-traditional way of executing a war. The way we run our exercises may not be representative of what the next fight looks like. So the more important focus should be on the pipeline, the processes, and the governance required for AI to be updated repeatedly, safely, and in a trustworthy way. Jordan Schneider: Whose job should this be? Is it something the government should outsource? Should they train their own models? Should it be the weapons manufacturers, or a whole separate data team? What’s the ideal setup? Bharat Patel: The ideal setup is that the government is responsible for managing the data and keeping it in the government’s hands. The pipeline, the process, and the governance are the government’s responsibility, but the work itself should be outsourced as much as possible. Government engineers should be doing government development so they’re intellectually aware, can keep up with the state of the art, understand industry’s ability to meet requirements, and have those conversations with industry. But industry should be doing about 90% of the work. There’s a data management part, a data labeling part, a test and evaluation part, a deployment part, a model training part — a ton of areas for industry to participate. It’s up to the government to break those into a modular approach that lets us continuously iterate with industry without ever getting locked into a single partner longer than needed. Acquisitions and Reforms Jordan Schneider: What is it like as an acquisitions official to try to stand something like this up? What constraints are you operating under? Bharat Patel: Oh man, all types — from personalities to policies to resources. But before we get into that, let me tell you how much fun I had as an acquisition professional. You mentioned I worked at MITRE. I started my career with the Army acquisition community in 2007, supporting the Program Executive Office for Intelligence, Electronic Warfare and Sensors. We were still in Iraq and Afghanistan doing things, and the portfolio was supporting it. I got to see all of that firsthand. Then I moved to Project Manager Distributed Common Ground System–Army around 2016 as a federal employee — but always within the acquisition community, always within the same PEO construct. That entire time, I fundamentally loved every day, every minute of it, because — I always call it “the show.” Any good movie has a producer or production lead, and as a PM you have so many parts you don’t own that you have to bring together to get the job done. To stand something up, I don’t own the requirement, so I had to work with our requirements owners to help shape what a pipeline requirement could be. Then I had to talk to the resources folks, the acquisition headquarters, and the contracting folks. Everybody has a say. Test and evaluation organizations have a say. All these pieces have to come together to create the program. Of course, industry has one of the biggest votes, because industry is either going to barf on the entire strategy or get excited. The way we originally broke up what we called Project Linchpin — now Product Manager AI Ops and Services — was to allow maximum industry participation in the pipeline. Those are some of the constraints and joys of standing up a program. Jordan Schneider: The way you describe it, the technical engineering fluency of the thing you’re building is maybe a quarter of what you need. The 4D chess of bureaucracy — getting the colors of money right, getting people aligned in the building — seems to be just as important, if not more so, than understanding the physics or engineering behind it. Bharat Patel: You have to be able to navigate the Pentagon. Everyone really does want to move things forward, but not everyone will jump on a singular idea and push it. Some people have their own opinions even after something’s been bought off on, and you still have to move it forward. So you have to figure out who those people are and how to bring them on early. It becomes a people, personality, and organizational-understanding game to really move the needle on future programs. Jordan Schneider: Let’s talk about the reforms we’ve seen over the past two years — Portfolio Acquisition Executives. What’s the vision behind that restructuring that Feinberg has put into the building, and what can and can’t it change? Bharat Patel: Acquisition reform has been around forever. For the 20 years I’ve been part of this community, it’s always been about acquisition reform. There have been so many attempts — from policies put in by the acquisition community to the Army standing up Army Futures Command and cross-functional teams to accelerate acquisition, and now the latest with the PAEs and CPEs. It’s just another means to accelerate the acquisition community, but not necessarily one that tackles the hard part: holding the program manager — who is actually authorized to deliver the capability — accountable, and then giving them the appropriate resources to get it done. Now you’ve got PAEs. What I see, at least for the next few months, is that nobody is exactly sure what needs to be authorized by the PAEs versus what can happen at the CPEs, the PMs, and the product manager level. All of that is still being worked out. In theory, once it’s worked out — because the PAEs have a little more pull from a requirements and contracting perspective — it’s supposed to accelerate the ability to change requirements and move faster. We’ll see how it plays out. Jordan Schneider: Another thing the department talks a lot about is making speed something we optimize for in acquisitions. What’s your vision for getting those incentives right? Bharat Patel: Start small. It starts with the humans who are actually motivated to get something done. If the humans on the acquisition team aren’t motivated, you’re never going to accelerate. I’m so tired of “we’ve got to go fast, we’ve got to go fast,” and then nobody ever really goes fast — because we have really good people, but they’re scattered. The first thing I’d do, if something is high priority, is make sure my highest-speed individuals are on that acquisition team. They will move mountains. I was able to move mountains, and I brought people in to help me move even further. But it goes back to having the right people with the right incentives. There’s a lot we need to do on the government side to properly incentivize and resource the people we want executing these big programs. Jordan Schneider: You also worked on fielding capabilities for people fighting in wars today. I’m curious for your reflections on wartime versus peacetime acquisitions. Bharat Patel: This is where it gets very interesting. Wartime quick-reaction capabilities pop up like that, and we can get them on contract very quickly, because some policies are just waived off. But in peacetime, there are a lot of federal acquisition regulations, competitive prototyping, and competitive source-selection activities you simply have to follow. A lot of that shifts when you’re doing QRCs — quick-reaction capabilities — and you have to field stuff that will save lives or deter our enemies and threats. Things happen much quicker in wartime, and there are a lot more resources behind it: not just your regular service budgets, but contingency operations resources that let you move a ton faster. Contracting happens faster. And suddenly there’s a lot less concern about modular system architectures or being able to swap out components. There’s more concern about moving fast — we’ll figure out the seams later, as long as we can save lives. Synthetic Data and Data Poisoning Jordan Schneider: Let’s come back to data. Synthetic data — people are excited about it, people are worried about it. How useful is it for the targeting, autonomy, and physical real-world stuff we’re talking about? Bharat Patel: My synthetic data journey has gone like this. It started around 2022 when I was supporting the TITAN program — Tactical Intelligence Targeting Access Node, the next-generation ground station capability. One of my hopes was to accelerate the AI. Before we even fielded some of these capabilities, I wanted to see if we could pre-train models. I partnered with the Small Business Innovation Research program — I was lucky to be the first to use the Direct-to-Phase-II concept — and awarded four Direct-to-Phase-II contracts to companies to generate synthetic data. The first thing the companies said was, “Hey, do you have representative data?” And I thought, “Oh crap.” So it goes back to data being very important. Synthetic data is really dependent on the actual data, so the companies can replicate it. You need enough representative data for synthetic data to work. There’s a lot of variation in maturity across our industry partners on synthetic data. When it comes to autonomous vehicles, commercial industry is investing a ton — NVIDIA is doing some things, and the technology is getting there. But applied to the sensors the Army has, that’s different from the commercial sensors on airplanes or new autonomous vehicles. Military sensors could be two or three years older, with completely different look angles, different biomes, different experiences. Giving a synthetic data company enough of that data to replicate those scenarios is hard. So we aren’t quite there yet on building AI and autonomy off purely synthetic data. Synthetic data augments operationally relevant data — the combination gets you a bit more performance. The way I like to use synthetic data is to figure out the edge cases you don’t have operationally relevant data for. Replicate those edge cases so you can see what the model does against weird situations you just haven’t seen before. Jordan Schneider: Another difference between training Tesla Autopilot and your tank model is that no one is walking around the street expressly trying to trip up a Tesla into driving into people. That’s not necessarily the case when you have a human being trying to trick a robot that’s trying to shoot them. Talk a little about the adversary dynamics when it comes to data and what folks should expect or be prepared for in the coming years. Is data poisoning something to be particularly worried about here? How would you define that phrase? Bharat Patel: In many ways, we should be worried about it 100%. There are two scenarios. There’s active poisoning happening right now. The open-source environment — open-source technologies, databases, and model frameworks — is out there to help accelerate AI. But there are also people contributing to the negative aspects, putting in bad data for fun. I sometimes equate it — not exactly, but depending on my mood — to being a jerk on Waze: “cops here, cops there,” even when they’re not, when I’m just being a jerk. I put it there, and now as part of the enterprise, that biased data is in there. So there’s either purposefully bad, biased data, or sometimes by accident, somebody mislabeled something or added the incorrect equation into the data. The models get trained against that, so they come out biased in some way. That’s the red-teaming aspect. Then, once we’re in production with these AIs fielded, it’s still fairly easy to put camouflage over something — or spray-paint graffiti over a target — and it no longer registers as a target, because the model has never seen a target with graffiti on it. There are a lot of ways to spoof these capabilities, especially in computer vision. Even in the electromagnetic spectrum, you can put out fake signals that replicate the size of a large enemy movement. All you need is a wagon with a bunch of sensors on it that acts like a military movement. Without visual confirmation, it looks like a military movement. Jordan Schneider: Given the mess all of that creates, what do benchmarks and test and evaluation need to look like for these things? Bharat Patel: There are a couple of aspects. One, we can’t overtest and over-evaluate because we need to actually get the model and the capabilities into the field so they can experience the environment. We have to limit how much T&E we conduct — while still making sure the models are safe and we understand the risk of deploying them. This model needs to be, say, 80%, because the enemy is going to have a vote, the environment is going to have a vote, and the warfighter using the model is going to have a vote. So 80% might be good enough for them. Not overly testing models is 100% one of the things we need to accomplish. The other part is partnering with our intelligence community to understand all the possible activities adversaries use to spoof models, then building that into the T&E process so we can build a robust model. If we understand the enemy is graffiti-ing their targets or platforms, maybe we generate that as synthetic data and inject it into the T&E and training dataset, so the capability can detect and overcome it. Jordan Schneider: You mentioned earlier you want to write a sequel to Pentagon Wars – the first one is on HBO Max. The challenges of doing test and evaluation on a Bradley fighting vehicle are relatively straightforward: here’s a range of temperatures, a range of ammunitions, it could be hit by this, that, or the other. But when you’re talking about what an autonomous vehicle in a fight, broadly defined, could face in the future, there are so many more variables — and surely things you can’t even come up with sitting here in 2026. How do you navigate that uncertainty when figuring out how much to trust these models in autonomous vehicles? Bharat Patel: One, we can’t hide from it. We’ve got to figure out a way to go do stuff safely. A lot easier said than done. AI is here, and we’ve got to start exploring and getting into the right environments where we can do some of these edge cases that aren’t necessarily safe or survivable. We’ve got to be able to do that. And two — yes, Pentagon Wars 2: The Race for AI. It’s going to be great. It’s a calamity — my journey, surrounded by really cool senior leaders and maybe some not-so-cool ones, of creating a program and all the great things that came with it: the conferences, flying on military air, talking to congressional appointees and senior leaders, building partnerships across the entire Department of War. It was a phenomenal experience. Transitioning that into how we’re going to do this — like I said, you’ve just got to go after it. We really do need to focus on the boring parts of AI: test and evaluation, verification, validation. When I was starting Project Linchpin, there was NDAA language about a study that concluded the department really needs to centralize on a standard for test and evaluation, because that’s how you scale AI. Once you understand how to ensure it’s reliable, you’ll be able to field in large production quantities, versus these one-offs where you can’t understand the capabilities and limitations, who’s using the AI, and what decisions and recommendations they’ll make based on its output. Jordan Schneider: Okay, you have your Bradley, and maybe every six months you’ll put new armor on it. But the promise and peril of being able to update the software on your robots daily or weekly is kind of the answer to that test and evaluation problem, right? Yes, you’ll run into things you weren’t predicting, but as long as you can capture that data, learn from it, and let the robots get smarter, that’s the answer — as opposed to trying to cover all your bases before the serious stuff starts. Bharat Patel: 100%. There are a couple of aspects. Way back when, the department decided to separate hardware and software. We’re now at a point where we’re ready to separate hardware, software, and AI. AI is really about data and models, whereas software is the application, the functions. If you can separate software from AI, you can speed up your AI process for rapid, repeatable capabilities. You don’t have to update your entire software package — just the model. Now we’re bleeding into the evolution of the RMF, the Risk Management Framework in the cyber process. Right now, cyber treats everything as software: you accredit the whole thing, then re-accredit the whole thing. But maybe we don’t need to re-accredit the entire software package — we just need an updated model. Before I left, the plan was to separate AI from software so that, from a cyber perspective, we could do that, and from a technology perspective, we could push out more updates. Jordan Schneider: Let’s close on some of the work you’re doing at Accenture. Why should AFS exist in the first place? What’s the role of these large federal contractors in helping create the stuff we’ve been talking about? Bharat Patel: I’ve been here about six months. I feel like I was blessed in my career — I have a pretty large network and could have gone almost anywhere I wanted. I selected Accenture for a couple of reasons. Right now, the department is 100% fascinated by products. But once the shiny-toy phase of products is over, the department is going to realize it needs a qualified integration team to make everything work. What Accenture does on a global scale is literally that: they integrate new technologies and leverage partnerships, physically integrating new technologies into Google, Caterpillar, Nike — these large integration efforts from enterprise to edge. I knew the department was eventually going to realize it needs qualified integrators, and when that happens, Accenture is going to be spot on. The other part is that Accenture is building a ton of partnerships — with OpenAI, Palantir, Anthropic, Databricks, all these bleeding-edge companies. We’re going to be well positioned to close the gap and meet requirements rapidly, because we have qualified engineers across all these technologies. Once we’re on contract, we’ll be able to help our government partners accelerate their capabilities. Jordan Schneider: Any specific projects you’re excited about? I hear you’re hiring as well. Bharat Patel: We’re hiring a ton. Accenture is at the cusp of blasting off. The part of Accenture we’re working in feels like one of the largest startups. We’re in a constant rat race: What’s next? How do we stay competitive? How do we stay in front of our competitors? How do we understand — and shape — what the government needs, because we’ve understood that environment for a while? We’re projecting some great things in the next two or three months, and then we’re going to need solid people to come on board and join the culture, join the team that’s changing the landscape for companies like us.