GitHub Copilot gains desktop automation, but company urges caution over APIs

GitHub unveiled computer use in public preview this week, extending Copilot CLI and its desktop application with the ability to navigate and control software running on macOS and Windows. The capability permits agents to perceive application interfaces and perform actions including clicking, typing, scrolling, and dragging—even within legacy software that lacks an API, command-line interface, or MCP integration.

During its announcement, GitHub demonstrated the feature using an expense report in Safari as a test case. The company outlined additional potential applications: pulling data from older systems, modifying presentation slides, populating forms, and transferring content across multiple programs. Users can trigger the feature either from a terminal or via the Copilot app, which debuted earlier this year as a competitor to Anthropic's Claude Code and OpenAI's Codex.

The rollout positions GitHub behind its rivals in the desktop automation space. OpenAI integrated computer use into Codex during April, while Anthropic deployed broader computer use capabilities to Claude Code and Claude Cowork earlier in the year.

Computer use versus MCP servers

Activating computer use within Copilot CLI engages a built-in plugin containing its own MCP server. The mechanism operates within local environments, extracting application content via the operating system's accessibility tree and capturing screenshots when visual information becomes necessary.

GitHub's guidance emphasizes prioritizing direct tools wherever feasible. When an API, MCP server, terminal command, filesystem tool, or specialized browser tool can accomplish the task, these approaches typically deliver more organized data and more consistent outcomes compared to simulated desktop interactions.

This recommendation reflects GitHub's view on the appropriate scope for computer use. OpenAI president Greg Brockman articulated a contrasting perspective recently, contending that agents could leverage the same interfaces humans use, thereby eliminating the necessity for the industry to construct and sustain integrations for every application.

Saved approvals outlast their removal

Users activate the capability by entering /computer on in Copilot CLI or navigating to Computer Use settings within the Copilot app. On macOS, the system additionally requires Accessibility permission to manipulate controls and Screen Recording permission to view windows when visual context becomes necessary.

The permission configuration in a CLI session dictates whether Copilot requests authorization before interacting with an application. Developers can review this setting using /permissions show. When a prompt appears, they may grant access for that specific session, select "Always allow" for subsequent sessions, or refuse. Deny rules take precedence over both automatic and previously saved approvals.

Approvals stored in the CLI transfer to the desktop app on the identical machine. Removing an application from the always-allowed list revokes its approval for upcoming sessions, though access already granted in an active session persists.

Halting operations requires deliberate action: press Esc twice in the CLI, or select Stop or press Esc in the desktop app.

Enterprise controls over computer use

Organizational policy supersedes individual developer settings. Should managed settings prohibit computer use, Copilot CLI indicates the feature is inaccessible.

Enterprise policy overrides a developer's local preference.

Using managed-settings.json, administrators can establish whether developers may skip approval prompts. This constraint spans the Copilot app, CLI, and VS Code.

GitHub's standard-enablement approach for Business and Enterprise tiers does not alter the preview's current opt-in requirement. The policy begins affecting unconfigured features on October 22, with the exception of preview features.

Reliability depends on the interface

Variations in timing or window conditions can trigger Copilot to duplicate an action or become unresponsive. GitHub cautions that the agent might select an incorrect control, enter text in an unintended location, or encounter difficulties with interfaces that change dynamically and intricate processes.

Sensitive information visible in an application window may also become context for the agent.

Unanticipated visual elements and vague directives can result in operations that alter the user's system, stored data, or linked services. Confidential data displayed within an application's window can serve as input for the agent's decision-making.