Cursor's Computer Use Lets Its Coding Agent Click Through Desktop Apps

retained text

Cursor's Computer Use Lets Its Coding Agent Click Through Desktop Apps
Cursor's local agent can now click and type across any desktop app, driving tools like the iOS Simulator to verify the code it writes.
- Cursor's in-editor agent gains Computer Use, controlling any desktop app on your local machine.
- Flagship demo: driving the iOS Simulator to tap through and verify features it just built.
- Trigger it by @-mentioning Computer Use, or let the agent invoke it automatically when needed.
- Previously available only in cloud agents' isolated VMs or self-hosted workers, not the local IDE.
- Closes a long-standing gap versus Claude Computer Use and OpenAI's Codex CUA.
- Requires updating the Cursor desktop app; no pricing change announced.
Cursor has added Computer Use to its desktop IDE, allowing the local coding agent to click, type, and capture screens across GUI apps on a developer’s machine. The feature extends agent workflows beyond source edits, shell commands, and browser DOM automation. An agent can now build an iOS feature, launch Simulator, tap through the flow, inspect the result, and revise the code within one task.
Developers can invoke Computer Use by adding @Computer Use to a prompt. Cursor may also select the tool automatically when a task requires interaction with a visual interface. The feature is available by updating the desktop app from the Cursor download page. Cursor announced no accompanying pricing change.
Cursor’s earlier local agent could edit files, run commands, and operate web pages through a browser tool, while native desktop workflows required manual input or external tool servers connected through the Model Context Protocol, known as MCP. In an earlier forum reply, the company said it was exploring local desktop control without providing a timeline.
The GUI joins the build loop
Computer Use connects the agent to the machine’s active display and input controls. Apps with accessible visual interfaces can become part of an automated development task, subject to operating-system permissions and application compatibility.
- The agent edits the project and runs the build.
- It opens a native app, simulator, installer, or system interface.
- It clicks controls, enters text, and navigates the workflow.
- It captures the resulting screen and evaluates the output.
- It changes the code and repeats the process when needed.
The agent can choose the tool automatically or follow an explicit mention in the prompt:
Cursor now has three desktop targets
Cursor already supported computer control in cloud agents and self-hosted workers. The new release adds the developer’s active desktop to that architecture.
Self-hosted workers use a worker helper that can click, type, and capture the worker’s screen. On a Mac worker, the capability starts with agent worker --computer-use start. Adding the user’s desktop to the IDE agent makes locally installed tools such as Xcode and Simulator available during the same session as code editing.
Why Simulator leads the demo
Mobile development exposes a persistent limitation in coding agents: the build often ends in a visual interface that requires taps, text entry, and navigation. Cursor’s earlier Render Preview could capture simulator screenshots for the agent, while interaction inside the simulated device still required manual input.
Computer Use can operate the full test flow:
- Build and launch the application.
- Open the iOS Simulator.
- Tap through onboarding or navigation paths.
- Enter text and submit forms.
- Capture screens for visual inspection.
- Revise the implementation and rerun the flow.
This workflow reduces handoffs during interface testing and gives the agent direct evidence about layout, navigation, and visible application state.
Useful beyond mobile
Computer Use fits development tasks where shell commands and browser selectors provide incomplete access to the interface. Likely targets include:
- Desktop applications without a practical command-line interface.
- Native clients, design tools, launchers, and installers.
- System dialogs and multi-window workflows.
- Visual regression checks that depend on rendered output.
- End-to-end tests spanning a browser, native app, and operating-system controls.
Desktop access widens the risk surface
An agent controlling the active desktop can encounter private messages, credentials, production accounts, destructive controls, and unrelated applications. It can also conflict with human mouse or keyboard input during a run. Teams adopting the feature need to treat GUI access as a privileged capability.
Cursor’s self-hosted path requires explicit opt-in, and macOS grants Accessibility and Screen Recording permissions to the helper application specifically, according to its self-hosted permissions documentation. Permission behavior, action confirmations, application restrictions, screenshot retention, and model routing remain important checks for the local IDE path. In this context, “local” identifies the controlled machine and does not establish on-device model execution.
Practical safeguards include:
- Use test accounts and non-production environments for automated flows.
- Close unrelated applications and remove sensitive information from the visible workspace.
- Commit or stash changes before allowing the agent to perform multi-step actions.
- Monitor workflows that can delete data, submit forms, publish content, or change system settings.
- Use a dedicated operating-system account or test machine when stronger isolation is required.
GUI control is becoming standard
Anthropic offers computer control through Claude, OpenAI has developed its Computer-Using Agent technology, and GitHub Copilot has introduced a desktop-control preview for macOS and Windows. Cursor previously covered remote and self-hosted desktop environments; Computer Use brings the same category of interaction into its local editor workflow.
For developers, the practical change is a longer autonomous loop. Cursor’s agent can move from implementation to native interface testing and back to code without requiring a person to operate each screen between those steps.

document fetched 2026-10-24 15:22