Vercel Labs releases open-source agent-desktop tool for AI agent...

Vercel Labs releases open-source agent-desktop tool for AI agent computer use

Vercel Labs has released agent-desktop, an open-source tool (Apache-2.0, written in Rust) that enables AI agents to control the desktop via the OS accessibility tree instead of screenshots, returning the interface as a JSON snapshot with a ref for each element.

Compiled by Sergey KostenchukPublished 2026-11-04Updated 2026-11-04

2026-11-04 Coding

🛠 Vercel Labs releases agent-desktop — computer use for AI agents

The Rust-based tool under the Apache-2.0 license operates not through screenshots, but via the OS accessibility tree: AX API on macOS, UI Automation on Windows, and AT-SPI on Linux. The interface is provided as a JSON snapshot: each element has its own ref, and the agent clicks and types using the ref, then re-reads the screen for verification.

🌍 Instead of pixel recognition, it offers deterministic handles on UI elements: the interface snapshot is text, and errors with a disposition.retry field distinguish safe retry attempts from dangerous ones.

👤 It can be installed via npm install -g agent-desktop or Homebrew. By default, everything works headless — without moving the cursor or intercepting focus, allowing users to continue working on the same computer. It includes a CLI, a C library, and an MCP mode.

Source 1: https://github.com/vercel-labs/agent-desktop

Source 2: https://www.harelasaf.com/articles/vercel-labs-agent-desktop-accessibility-tree-computer-use-retry-contract

Sources