Computer use
Screen and input tools for native apps on macOS, Windows, and Linux.
Computer use lets the agent look at your screen and operate native applications and browsers outside pi-kit. It is provided by the bundled
@pi-kit/computer-use package, which runs Cua Driver inside the pi process. It is off until you turn it on.Supported platforms#
| Platform | Architectures | Setup |
|---|---|---|
| macOS | Apple Silicon | Grant Accessibility and Screen & System Audio Recording. |
| Windows | x64, ARM64 | No permissions needed. |
| Linux | x64, ARM64 (glibc) | No permissions needed. Requires a graphical X11 or Wayland session. |
pi-kit installs the package into pi once on first start. The release builds carry the matching Cua Driver library for each platform; nothing else needs to be installed.
Turn it on#
Open Settings → Computer use and switch on Let pi use this computer, or type
/computer-use on in the composer.| Command | Effect |
|---|---|
/computer-use on | Asks once, then gives pi the computer tools. |
/computer-use off | Removes the tools. |
/computer-use setup | macOS: opens each permission page in turn. |
/computer-use status | Shows whether it is on and whether permissions are ready. |
The setting is one saved choice shared by pi-kit and the pi command line.
macOS permissions#
macOS grants Accessibility and Screen & System Audio Recording to the application that started pi, not to the package. Settings → Computer use shows both permissions with live state, an Open System Settings button for each, and which application macOS will ask about: pi-kit for the release build, Electron for a development build, or your terminal when the backend was started from one.
macOS applies a new Screen Recording grant only after the app restarts; use Relaunch when it is offered. While a permission is missing, threads show a Set up computer use permissions pill that opens this page.
An ad-hoc signed development build loses its permissions whenever it is rebuilt. The signed release build keeps them.
What the agent can do#
The agent receives Cua Driver's tools, named
computer_…: listing windows, reading window state, taking screenshots, clicking, typing, and pressing keys. Screenshots return as images, so interpreting them requires a model that accepts image input.For example:
Open the Calculator app, compute 17 × 23, and tell me the result shown on screen.
Approvals#
- Looking (listing apps, reading a window, taking a screenshot) runs without asking.
- Acting (clicking, typing, pressing keys) asks once per session.
- Running a script, quitting an app, downloading or saving something asks every time.
Questions appear as Allow / Deny blocks in the thread. A run that cannot ask, such as print or JSON mode in the pi CLI, cannot act.
Safety and limits#
Anything on screen is untrusted input: a web page or document can contain instructions aimed at the agent. Keep sensitive windows closed while computer use is active and review each approval.
The Cua Driver SDK is pre-1.0 and pinned to a tested version. On Linux, a session without a display (for example SSH without a desktop) has nothing to look at. Windows and Linux support has been packaged and smoke-tested, but has not yet been exercised on every desktop environment.