#Computer Use

computer-use is a worker process built to drive a real desktop:

• Mouse and keyboard
• Screenshots
• Clipboard
• Open windows
• The accessibility tree

It runs as a process separate from aiod, on any host with a desktop. Examples:

• An Ubuntu desktop
• A VM with Xvfb or Xvnc
• A Windows host with an interactive logon session

aiod proxies its API. It does not touch the desktop itself. That process boundary keeps desktop control optional: aiod runs the same without a desktop present, and reaches one wherever computer-use is running.

The desktop plane is v2-only, under /v2/computer/*.

v1 keeps two aliases for 1.x clients:

• POST /v1/browser/actions
• POST /v1/display/record

The routes side by side are in Migration from 1.x.

The first part of this page is the action reference an agent's tools are built on. The second covers the viewers a person uses to watch or take over the same desktop. Browser API covers CDP-based page automation instead, and Computer Use examples walks one desktop task end to end.

#For agent tools

#How the two processes connect

The worker binds AIO_COMPUTER_USE_LISTEN (default 0.0.0.0:18100). aiod reaches it at AIO_COMPUTER_USE_URL (default http://127.0.0.1:18100) and proxies the desktop routes opaquely. When the worker is not running, the computer routes answer 503:

{
  "success": false,
  "message": "computer-use worker is not available",
  "data": null,
  "hint": "start computer-use or set AIO_COMPUTER_USE_URL"
}

#Prerequisites

Linux: a reachable X11 display. Set DISPLAY, and XAUTHORITY if the display needs cookie authentication. GET /v2/computer/info reports what it found in display and xauthority.

Windows: an interactive logon session. aiod can run as a Session 0 SCM service; computer-use must run in the interactive session to see the desktop and inject input.

#Start the worker

computer-use ships with every aiod release; see Releases. On Linux the install script installs both:

curl -fsSL https://aio-static.tos-cn-beijing.volces.com/install.sh | AIOD_WITH_COMPUTER_USE=1 sh

On Windows download https://aio-static.tos-cn-beijing.volces.com/latest/windows-x86_64/computer-use.exe.

It takes no flags and reads only environment variables: DISPLAY and XAUTHORITY pick the display, AIO_COMPUTER_USE_LISTEN sets the listen address.

Linux:

DISPLAY=:99 XAUTHORITY=$HOME/.Xauthority computer-use
curl -s http://127.0.0.1:8091/v2/computer/info

supervisord, as the Computer image runs it:

[program:computer-use]
command=/usr/local/bin/computer-use
environment=DISPLAY=":99",XAUTHORITY="/home/sandbox/.Xauthority",HOME="/home/sandbox",USER="sandbox"
user=sandbox
autorestart=true
stdout_logfile=/var/log/computer-use.log
redirect_stderr=true

Windows: computer-use.exe cannot run as a service; register it as a logon-triggered scheduled task, see Windows.

#Check available capabilities

curl "$BASE_URL/v2/computer/info"

curl "$BASE_URL/v1/capabilities"

{
  "status": "absent",
  "provider": null,
  "display": null,
  "screenshot": false,
  "actions": false,
  "clipboard": false,
  "recording": false,
  "accessibility": false,
  "accessibility_backend": null,
  "resolution": {
    "width": null,
    "height": null
  },
  "missing": [
    "computer-use"
  ],
  "warnings": [
    "computer-use worker probe failed: error sending request for url (http://127.0.0.1:18100/capabilities)"
  ]
}

#Take a screenshot

curl "$BASE_URL/v2/computer/screenshot" -o desktop.png

curl -X POST "$BASE_URL/v1/browser/actions?include_screenshot=true" \
  -H "Content-Type: application/json" \
  -d '{"action_type": "WAIT", "duration": 0.5}'

#Act

One action per request, in screen coordinates. With ?include_screenshot=true the response also carries a screenshot taken after the action:

curl -X POST "$BASE_URL/v2/computer/actions" \
  -H "Content-Type: application/json" \
  -d '{"action_type": "CLICK", "x": 640, "y": 400}'

curl -X POST "$BASE_URL/v2/computer/actions?include_screenshot=true" \
  -H "Content-Type: application/json" \
  -d '{"action_type": "SCROLL", "dx": 0, "dy": -3}'

curl -X POST "$BASE_URL/v1/browser/actions" \
  -H "Content-Type: application/json" \
  -d '{"action_type": "CLICK", "x": 640, "y": 400}'

curl -X POST "$BASE_URL/v1/browser/actions?include_screenshot=true" \
  -H "Content-Type: application/json" \
  -d '{"action_type": "SCROLL", "dx": 0, "dy": -3}'

from agent_sandbox import Sandbox
from agent_sandbox.browser.types.action import Action_Click

client = Sandbox(base_url="http://127.0.0.1:18091")
client.browser.execute_action(request=Action_Click(x=640, y=400))

import { SandboxClient } from "@agent-infra/sandbox";

const client = new SandboxClient({ environment: "http://127.0.0.1:18091" });
await client.browser.executeAction({ action_type: "CLICK", x: 640, y: 400 });

The action body is the same in v1 and v2. For example, click screen coordinates (640, 400):

{
  "action_type": "CLICK",
  "x": 640,
  "y": 400
}

The body describes the action directly, and action_type selects which action to run:

• duration is in seconds; keys is a list, ["ctrl", "c"] for copy.
• A window_id comes from the window list, a node_id from the accessibility tree; both are below.
• Two bounds hold on every action, single or batched: a WAIT asks for at most 10 s, and a SCROLL for at most 100 notches per axis.
• TYPING types ASCII text as keystrokes. Text with other characters goes through the clipboard (use_clipboard, default true), which replaces whatever the clipboard held; with use_clipboard: false such text is refused with 400, since keystrokes would drop the characters. Clipboard text is pasted with Shift+Insert, which terminals and GTK or Chromium windows both accept.
• A successful PRESS, HOTKEY, or TYPING means the input reached the focused window, not that the application acted on it. Take a screenshot to confirm.
• The full per-type schema is in the API Reference.

The response carries status and action_performed, plus the post-action frame as base64 PNG when one was asked for. If capturing it fails, that shows up as screenshot_error next to the action result, not as a failed request: the action already happened.

A Windows secure desktop (a UAC prompt, the lock screen) refuses input; see Windows.

#Batch several actions

curl -X POST "$BASE_URL/v2/computer/actions/batch" \
  -H "Content-Type: application/json" \
  -d '{"include_screenshot": true,
       "actions": [
         {"action_type": "HOTKEY", "keys": ["ctrl", "l"]},
         {"action_type": "TYPING", "text": "example.com"},
         {"action_type": "PRESS", "key": "enter"}
       ]}'

#Run your own automation

The desktop is an ordinary X display and every command runs with DISPLAY set, so a script that drives the screen itself runs through POST /v2/commands (/v1/bash/exec on v1) against the same desktop the actions above use. The Computer image ships xdotool; pyautogui is one pip install pyautogui away:

sb = Aio(BASE_URL)  # the helper on the examples index
sb.post("/v2/commands", command="pip install --quiet pyautogui", timeout=280)
sb.post("/v2/commands", command='''python3 - <<'EOF'
import pyautogui
pyautogui.moveTo(320, 240)
pyautogui.click()
pyautogui.write("hello")
pyautogui.screenshot("/tmp/desk.png")
EOF''')
print(sb.get("/v2/computer/cursor"))   # {'x': 320, 'y': 240}

#Read the pointer, clipboard, and windows

{snapshot_id, windows: [{window_id, title, process_id, bounds, minimized}]}

#Act through the accessibility tree

{
  "node_id": "a::1.30:/org/a11y/atspi/accessible/247",
  "role": "link",
  "name": "Open the report"
}

{
  "action_type": "NODE_INVOKE",
  "node_id": "a::1.30:/org/a11y/atspi/accessible/247"
}

Backends by platform: Windows uses UI Automation (UIA), built into the OS with nothing to install; Linux uses AT-SPI2 (docs).

Three things make a Linux desktop readable as a tree:

1. at-spi2-core installed: it provides the org.a11y.Bus D-Bus service and the bus launcher.
2. A session D-Bus the worker finds through DBUS_SESSION_BUS_ADDRESS.
3. Applications that expose their own tree. GTK and Qt apps do by default; Chromium and Electron need --force-renderer-accessibility.

The Computer image ships at-spi2-core, starts a session D-Bus, and runs Chromium with that flag.

Errors:

#Record the desktop

client.display.record(action="start", save_path="/workspace/recordings/session.mp4")
client.display.record(action="stop")

await client.display.record({
  action: "start",
  save_path: "/workspace/recordings/session.mp4",
});
await client.display.record({ action: "stop" });

Recording uses ffmpeg, and only one recording runs at a time. start takes the capture settings:

save_path is optional; without one the file lands in the temp directory as recording_<timestamp>.mp4. A missing parent directory is created, and one the worker cannot write to is a 400.

data carries:

• status — recording, stopped, or idle
• duration — seconds recorded so far
• save_path — where the file is being written
• file_size_bytes — the size after a stop

Store recordings under a path the File API can reach. Stop recording before tearing down the host, so the file stays playable.

#Talking to the worker directly

computer-use also answers on its own port, without going through aiod:

Useful when deploying or health-checking the worker on its own.

#Human in the loop

A person steps into the same terminal, browser or desktop the agent is driving: to watch a run, to take over a login or a captcha, to hand the desktop back. Nothing in the daemon changes for that. The viewers are pages the prebuilt images serve through their gateway, and each one attaches to the object the agent already uses:

Xvnc is both the X server and the VNC server, so a person's clicks land on the display directly and the worker sees the result in its next screenshot. The daemon serves the API only; aiod alone answers 404 on every path above.

The prebuilt images wire this up. The AIO image runs Xvnc on DISPLAY=:99.0 with openbox and Chromium, and serves noVNC, DevTools and the WebShell. The Computer image sets AIO_DESKTOP=xfce on top of it: an XFCE session takes the display, the computer-use worker starts next to aiod, a session D-Bus serves AT-SPI, and Chromium runs with --force-renderer-accessibility so tab contents appear in the accessibility tree.

#Related

• Computer Use examples — one verified run: navigate, follow a link through the tree, observe
• Browser API — CDP-based page automation
• File Operations — fetch recordings and downloads