#Computer Use

computer-use is a worker process built to drive a real desktop:

• Mouse and keyboard
• Screenshots
• Clipboard
• Open windows
• The accessibility tree

It runs as a process separate from aiod, on any host with a desktop. Examples:

• An Ubuntu desktop
• A VM with Xvfb or Xvnc
• A Windows host with an interactive logon session

aiod proxies its API. It does not touch the desktop itself. That process boundary keeps desktop control optional: aiod runs the same without a desktop present, and reaches one wherever computer-use is running.

The desktop plane is v2-only, under /v2/computer/*.

v1 keeps two aliases for 1.x clients:

• POST /v1/browser/actions
• POST /v1/display/record

The routes side by side are in Migration from 1.x.

The first part of this page is the action reference an agent's tools are built on. The second covers the viewers a person uses to watch or take over the same desktop. Browser API covers CDP-based page automation instead, and Computer Use examples walks one desktop task end to end.

#For agent tools

#How the two processes connect

The worker binds AIO_COMPUTER_USE_LISTEN (default 0.0.0.0:18100). aiod reaches it at AIO_COMPUTER_USE_URL (default http://127.0.0.1:18100) and proxies the desktop routes opaquely. When the worker is down, every one of them answers 503:

{
  "success": false,
  "message": "computer-use worker is not available",
  "data": null,
  "hint": "start computer-use or set AIO_COMPUTER_USE_URL"
}

Both v1 aliases answer the same way, since they reach the same worker.

#What the worker needs

Linux: a reachable X11 display. Set DISPLAY, and XAUTHORITY if the display needs cookie authentication. GET /v2/computer/info reports back what it found in display and xauthority.

Windows: an interactive logon session. aiod itself can run as a Session 0 SCM service.

computer-use must run inside the interactive session to see the desktop and inject input.

#Check available capabilities

curl "$BASE_URL/v2/computer/info"

curl "$BASE_URL/v1/capabilities"

{
  "status": "absent",
  "provider": null,
  "display": null,
  "screenshot": false,
  "actions": false,
  "clipboard": false,
  "recording": false,
  "accessibility": false,
  "accessibility_backend": null,
  "resolution": {
    "width": null,
    "height": null
  },
  "missing": [
    "computer-use"
  ],
  "warnings": [
    "computer-use worker probe failed: error sending request for url (http://127.0.0.1:18100/capabilities)"
  ]
}

#Take a screenshot

curl "$BASE_URL/v2/computer/screenshot" -o desktop.png

curl -X POST "$BASE_URL/v1/browser/actions?include_screenshot=true" \
  -H "Content-Type: application/json" \
  -d '{"action_type": "WAIT", "duration": 0.5}'

#Act

One action per request, addressed in screen coordinates. ?include_screenshot=true merges the observe step into the act step:

curl -X POST "$BASE_URL/v2/computer/actions" \
  -H "Content-Type: application/json" \
  -d '{"action_type": "CLICK", "x": 640, "y": 400}'

curl -X POST "$BASE_URL/v2/computer/actions?include_screenshot=true" \
  -H "Content-Type: application/json" \
  -d '{"action_type": "SCROLL", "dx": 0, "dy": -3}'

curl -X POST "$BASE_URL/v1/browser/actions" \
  -H "Content-Type: application/json" \
  -d '{"action_type": "CLICK", "x": 640, "y": 400}'

curl -X POST "$BASE_URL/v1/browser/actions?include_screenshot=true" \
  -H "Content-Type: application/json" \
  -d '{"action_type": "SCROLL", "dx": 0, "dy": -3}'

from agent_sandbox import Sandbox
from agent_sandbox.browser.types.action import Action_Click

client = Sandbox(base_url="http://127.0.0.1:18091")
client.browser.execute_action(request=Action_Click(x=640, y=400))

The action body is the same in v1 and v2. For example, click screen coordinates (640, 400):

{
  "action_type": "CLICK",
  "x": 640,
  "y": 400
}

The body describes the action directly, and action_type selects which action to run:

• duration is in seconds; keys is a list, ["ctrl", "c"] for copy.
• A window_id comes from the window list, a node_id from the accessibility tree; both are below.
• Two bounds hold on every action, single or batched: a WAIT asks for at most 10 s, and a SCROLL for at most 100 notches per axis.
• TYPING types ASCII text as keystrokes. Text with other characters goes through the clipboard (use_clipboard, default true), which replaces whatever the clipboard held; with use_clipboard: false such text is refused with 400, since keystrokes would drop the characters. Clipboard text is pasted with Shift+Insert, which terminals and GTK or Chromium windows both accept.
• A successful PRESS, HOTKEY, or TYPING means the input reached the focused window, not that the application acted on it. Take a screenshot to confirm.
• The full per-type schema is in the API Reference.

The response carries status and action_performed, plus the post-action frame as base64 PNG when one was asked for. If capturing it fails, that shows up as screenshot_error next to the action result, not as a failed request: the action already happened.

#Batch several actions

curl -X POST "$BASE_URL/v2/computer/actions/batch" \
  -H "Content-Type: application/json" \
  -d '{"include_screenshot": true,
       "actions": [
         {"action_type": "HOTKEY", "keys": ["ctrl", "l"]},
         {"action_type": "TYPING", "text": "example.com"},
         {"action_type": "PRESS", "key": "enter"}
       ]}'

#Run your own automation

The desktop is an ordinary X display and every command runs with DISPLAY set, so a script that drives the screen itself runs through POST /v2/commands (/v1/bash/exec on v1) against the same desktop the actions above use. The Computer image ships xdotool; pyautogui is one pip install pyautogui away:

sb = Aio(BASE_URL)  # the helper on the examples index
sb.post("/v2/commands", command="pip install --quiet pyautogui", timeout=280)
sb.post("/v2/commands", command='''python3 - <<'EOF'
import pyautogui
pyautogui.moveTo(320, 240)
pyautogui.click()
pyautogui.write("hello")
pyautogui.screenshot("/tmp/desk.png")
EOF''')
print(sb.get("/v2/computer/cursor"))   # {'x': 320, 'y': 240}

#Read the pointer, clipboard, and windows

#Act through the accessibility tree

Backends: AT-SPI2 (Assistive Technology Service Provider Interface) on Linux, UI Automation (UIA) on Windows. 501 means there is no backend to run here: a platform with no implementation, or a Linux image without the AT-SPI runtime. Retrying will not change that.

503 means a backend is there but could not read right now — no interactive desktop, or no focused window. Chromium and Electron apps expose their tree only when started with --force-renderer-accessibility.

#Record the desktop

client.display.record(action="start", save_path="/workspace/recordings/session.mp4")
client.display.record(action="stop")

Recording is ffmpeg-based; only one runs at a time. start takes the capture settings:

save_path is optional; without one the file lands in the temp directory as recording_<timestamp>.mp4. A missing parent directory is created, and one the worker cannot write to is a 400.

data carries:

• status — recording, stopped, or idle
• duration — seconds recorded so far
• save_path — where the file is being written
• file_size_bytes — the size after a stop

Store recordings under a path the File API can reach. Stop recording before tearing down the host. That keeps the file playable.

#Windows: the secure desktop

Input on a secure desktop is refused with a 403 whose data is {"status": "denied", "reason": "secure-desktop-or-uipi"}. Examples: a UAC prompt, the lock screen. This is a Windows session boundary, not a bug to work around. In a batch the same refusal arrives as status and reason inside a 200, because the actions before it really ran.

#Talking to the worker directly

computer-use also answers on its own port, without going through aiod:

Useful when deploying or health-checking the worker on its own.

#Human in the loop

A person steps into the same terminal, browser or desktop the agent is driving: to watch a run, to take over a login or a captcha, to hand the desktop back. Nothing in the daemon changes for that. The viewers are pages the prebuilt images serve through their gateway, and each one attaches to the object the agent already uses:

Xvnc is both the X server and the VNC server, so a person's clicks land on the display directly and the worker sees the result in its next screenshot. The daemon serves the API only; aiod alone answers 404 on every path above.

The prebuilt images wire this up. The AIO image runs Xvnc on DISPLAY=:99.0 with openbox and Chromium, and serves noVNC, DevTools and the WebShell. The Computer image sets AIO_DESKTOP=xfce on top of it: an XFCE session takes the display, the computer-use worker starts next to aiod, a session D-Bus serves AT-SPI, and Chromium runs with --force-renderer-accessibility so tab contents appear in the accessibility tree.

#Related

• Computer Use examples — one verified run: navigate, follow a link through the tree, observe
• Browser API — CDP-based page automation
• File Operations — fetch recordings and downloads