loading
loading
Mobile app testing is often a race against change. A redesigned screen, a renamed button or an unexpected pop-up can break an automated test, forcing developers to spend time fixing scripts instead of testing functionality. Bengaluru-based DrizzVision wants AI to make that process less brittle. Its agent, Fathom, uses visual models to interpret app screens and execute tests from natural-language instructions, rather than relying solely on predefined element identifiers. But making AI interact with an app is easier than making it do so reliably. DrizzVision's benchmark shows a sharp drop in performance across testing environments, underscoring the challenge of getting AI to navigate real-world apps consistently. A problem that began with testing The idea for DrizzVision grew out of a familiar frustration in mobile software development. "We would develop fast and then there would be two to three days of end to end testing happening," said Yash, one of the company's co-founders. Asad, another co-founder, had seen a similar problem from the business side. The two met in 2024, when vision models were beginning to demonstrate their ability to interpret computer screens. Yash began experimenting with these models to explore whether they could be used for software testing. Beyond code-based testing Most mobile automation tools rely on an application's underlying structure to identify elements. Tools such as Appium let testers write scripts that locate buttons using internal identifiers and perform actions like tapping or entering text. But software changes frequently. A renamed button or redesigned screen can cause an existing test to fail, even when the application's functionality remains intact. For companies managing thousands of test cases, the codebase can become "extremely large", Yash said, making tests difficult to maintain. DrizzVision takes a different approach. "We, on the other hand, just understand the screen and execute," Yash said. Rather than depend entirely on predefined element identifiers, the system interprets what is visible on the screen and acts on that basis. When an app does not follow the script Consider a food delivery app. A user placing an order might encounter a promotional offer, a permission request or another pop-up. DrizzVision wants its AI to recognise these interruptions and continue with the intended task. "A pop-up ideally should not break your test case," Yash said. "It should say that your checkout functionality is working fine. Although there were three pop-ups that we saw on our way." The same principle applies when interface elements change. If a button labelled "Edit" is replaced by a pencil icon, the AI should still recognise the function and continue. "Even in that situation it should self heal," Yash said. The company says it has reduced test flakiness, measured on an internal benchmark, from between 15% and 50% to around 3%. However, it cautions that results vary by client and use case. The limits of seeing a screen DrizzVision tested Fathom on Android across six frontier model configurations and 20 tasks in two environments. DrizzVision ran each task-model combination five times. In the easier suite, which used a standard Android image, the models passed 217 of 300 scored task runs, a pass rate of 72.3%. The harder suite used a rooted Android image, allowing the grader to inspect an application's database rather than rely on what appeared on the screen. The tasks were also designed to be more difficult. On this separate set of tasks, the models passed 86 of 300 scored runs, a pass rate of 28.7%. The two suites used different task sets and device images. DrizzVision said the image used for the first suite has since been retired, preventing the company from rerunning that suite. Performance also varied considerably across models, and the rankings changed between the two suites. GPT 5.6 Luna scored 64% in the easier suite but 4% in the harder one. Gemini 3.5 Flash scored lowest in the first suite at 54% but achieved 34% in the second, outperforming every GPT configuration tested there. DrizzVision cautions against interpreting these results as a definitive measure of AI capability. Its benchmark measures the complete system, including the model, Fathom's agent scaffold, and the specific Android image, not the model alone. There are also unresolved questions about the harder suite. Five of its 10 tasks scored zero across all six model configurations. DrizzVision said it verified each task's setup on a device before running the tests and did not believe broken tests caused the results. However, the zero scores remain unexplained without a task-by-task review. The company has not quantified how much they affect the overall pass rate. The results do not establish that AI cannot test applications. They do, however, show how much performance can depend on the task, the environment and the system used to execute it. Reliably navigating real applications remains a challenge. Keeping sensitive data on the device Visual testing also raises data security questions, particularly when applications contain personal or financial information. "The screenshot should not leave the device," Yash said. For client deployments, he said, DrizzVision deploys its own models on-premise rather than relying entirely on large frontier models running in the cloud. Yash described a model with fewer than one billion parameters that could handle half or more of the workload as a near-term milestone. Such a model could reduce dependence on larger models, although the company has not established how much of the workload it can handle in production. From a command-line tool to a wider market DrizzVision's first product was a command-line tool. The company said it piloted the tool with Walmart, allowing users to write tests in plain-text files and run them on local emulators. The product is now available through four routes: desktop, web, continuous integration and continuous delivery (CI/CD) pipelines, and a command-line tool with a model context protocol (MCP) interface for terminal-based use and AI coding assistants. The MCP interface works differently from Fathom's own agent. In this setup, the assistant's model handles the reasoning and uses DrizzVision's device tools to interact with the phone. Fathom itself runs in the desktop application. In the US, Yash said most customers come from Maestro, an open-source mobile testing tool. He expects DrizzVision to complement Maestro more often than replace it. Maestro uses YAML-based test flows, automatic waiting and retries, and the app's accessibility tree to locate elements, Yash said. DrizzVision's approach addresses a different problem: tests that break when interface elements or layouts change. Other companies are developing tools that generate tests from product documents, design files or Jira tickets. DrizzVision argues that generating tests is only part of the problem. Specifications describe what a product is supposed to do, not necessarily how the application behaves in practice. Tests that still depend on element identifiers can leave the maintenance burden unchanged. The company said it could support test generation from design files, provided the resulting tests run against the rendered application rather than relying solely on the design specification. Verifying AI agents could be a next market DrizzVision has raised $2.7 million in seed funding and has around 15 employees. The company said it has more than 10 enterprise customers and more than 20 small and medium-sized business customers in India. It has permission to name Tata 1mg as a customer. Yash sees a market beyond conventional software testing as AI agents begin interacting with applications on users' behalf. DrizzVision could eventually help verify whether these agents complete tasks correctly and safely. He believes contracts for this work could eventually be seven to 10 times larger than the company's current testing deals. However, he described this as a directional estimate. DrizzVision does not currently sell an AI agent verification service. For now, that remains a potential market rather than an established revenue stream. The company's immediate challenge is to demonstrate that visual AI testing can deliver consistent results across the varied conditions in which real applications operate.
document fetched 2026-10-25 02:57