# Computer use

> The experimental, off-by-default capability that drives your host desktop by screenshot and mouse/keyboard: platform support, the enable flow, OS permissions, the action loop, and the sandbox-derived safety model.

Computer use lets the agent operate your actual desktop — moving the real mouse, typing on the real keyboard, and reading the screen from screenshots — one action at a time. It exists for the last-resort case: a graphical app with no API, no command line, and no way in except the way a person would use it. It is **experimental and off by default**, and it is the most dangerous thing Fermix can do: it acts on your live, logged-in session, there is no undo, and the screenshots it reads are treated as untrusted content (a page could try to talk the agent into doing something). Turn it on only when you want the agent to drive the actual machine.

This is **host-desktop control**, not web automation. For clicking around a website or a JavaScript-rendered page, the separate `browser` tool drives a managed Chrome/Chromium process and is the right tool — see [capabilities and tools](/docs/capabilities-and-tools). Computer use is for the desktop itself: native apps, system dialogs, anything outside a browser tab. There is no "browser mode" here.

## How it works

The capability is a single tool, `computer_use`, that performs **one action per call**. The loop is always the same: take a `screenshot` to see the current state, then act on what you saw — click, type, press a key, scroll, or drag — using pixel coordinates read from that screenshot. Every action that changes something returns a **fresh screenshot** so the agent can check the result and retry a missed click, rather than acting blind. Because each step is verified against a new picture of the screen, the agent self-corrects as it goes.

## Supported platforms

Computer use ships a native helper binary — [compux](https://github.com/tezra-io/compux), an open-source Rust sidecar — and that binary exists only for two targets:

| Platform | Supported |
|---|---|
| **Apple Silicon macOS** (M-series) | Yes |
| **Linux x86_64, X11 session** | Yes |
| Intel Macs | No — install fails loudly |
| Linux on Wayland | No — capture and input injection are blocked; use an X11 session |
| Everything else | No — install fails with a clear "no build for this machine" error |

There is no silent degradation: an unsupported machine refuses the install with a typed error naming the target, rather than half-working.

## Enabling it

Turning computer use on is **one operator step** — set the feature flag in `~/.fermix/config.toml` and restart:

```toml
[fermix_core.computer_use]
enabled = true
```

The change applies on the **next daemon restart** (`fermix restart`). On a real daemon boot, before the readiness checks run, Fermix downloads the native helper matching that build if it is not already present. The helper is a small standalone binary that drives the desktop — it registers no tools of its own. It is fetched from the [compux](https://github.com/tezra-io/compux) project's own signed release and verified against a SHA-256 checksum baked into your Fermix build, then cached under `~/.fermix/plugins/compux/`. `fermix plugins` does not manage it. (This one component is delivered as a checksum-verified download rather than through the signed plugin-tarball pipeline the other catalog plugins use — see [plugins](/docs/plugins).)

That automatic fetch is deliberately narrow: it happens on **daemon boot only** — not during `fermix setup` — it is capped at about 30 seconds so a slow network cannot hold up the daemon, and it **fails soft**. If the download does not land, computer use stays off and you get it back on the next restart or by enabling it from the setup card. It exists so that an upgrade which moves the pinned helper version does not leave computer use silently dead until you notice.

You can also do it from the setup **Plugins** page: open the **Computer Use** card and enable it there, and Fermix writes the flag and fetches the helper for you.

The `computer_use` tool becomes visible to the model **only once both hold** — the feature is enabled *and* the helper is installed. A daemon that has one but not the other never advertises the tool at all, so the model cannot try to use a capability that is not actually ready.

On the setup Plugins page, once the helper is installed its card shows the feature's own readiness — **Ready** when it is enabled and installed, or **Needs setup** otherwise — instead of a generic plugin enable/disable state.

## Operating-system permissions

Driving the desktop needs the operating system's blessing, and this is deliberately a **diagnostic, not part of the enable gate**. The reason is a specific, nasty failure mode:

> On macOS, taking screenshots needs **Screen Recording** permission, but moving the mouse and typing needs **Accessibility** permission — and they are granted separately. If Screen Recording is granted but Accessibility is not, the agent can see the screen perfectly while every click and keystroke is **silently dropped**. It looks like the tool can only look and never act.

Because screenshots keep working without the input grant, a missing permission must *inform* the operator rather than hide the tool — which is why permission state is not folded into the "is the tool ready?" check. To see where you stand, run:

```bash
fermix doctor
```

and read its **computer use** line. It probes the helper without triggering a permission prompt and reports Screen Recording and Accessibility **distinctly**, with the exact fix. On macOS the fix it names for the silent-click trap is **System Settings → Privacy & Security → Accessibility**; for missing capture it names **Screen Recording** in the same pane. On Linux it reports whether you are on X11 (supported) or Wayland (not).

On macOS the helper runs as its own Developer-ID-signed, notarized `Fermix.app` — a permanent bundle identity of its own, rather than one inherited from the daemon build — so a Screen Recording or Accessibility grant sticks across upgrades and appears as a single **Fermix Computer Use** entry in System Settings, instead of re-prompting and leaving a new row behind every time you upgrade. The plain **Fermix** row you may also see in those panes is a different app — the [FermixPet voice companion](/docs/realtime-voice). The two share an icon on purpose, so only the name tells them apart. On the setup **Doctor** page, running the probe reports the same permission state; there, when a grant is actually missing on macOS, it also offers a **Grant macOS permissions** button that raises the Screen Recording and Accessibility prompts — registering the app with the OS so it shows up in System Settings — up front, rather than surprising you on the agent's first screenshot.

## The action loop

`computer_use` takes an `action` plus whatever arguments that action needs (coordinates, text, a key chord). Actions split into read-only actions, which never change the screen and always run, and mutating actions, which do change it:

| Kind | Actions |
|---|---|
| **Read-only** (always allowed) | `screenshot`, `mouse_move`, `wait`, `wait_for_change`, `inspect`, `elements` |
| **Mutating** | `left_click`, `right_click`, `double_click`, `left_click_drag`, `scroll`, `type`, `paste`, `key` |

A few of these are worth calling out:

- **`screenshot` with a `region`** zooms. Pass a rectangle `{x, y, w, h}` in the current screenshot's pixel space and you get back a **magnified crop** of just that area. Then pass the *same* region on the follow-up click or drag and give coordinates read from the magnified image — Fermix maps them back to the real pixel. This makes clicks on small or dense targets land far more accurately than guessing at full-screen coordinates.
- **`inspect`** (read-only, macOS) reports the accessibility element under a point — its role and label, e.g. "a button titled *Delete*" — so the agent can confirm it is about to click the right control before a consequential action. It informs the agent's own judgment; it is not a blocking gate.
- **`elements`** (read-only, macOS) lists the interactive elements on screen, each with a click point, so the agent targets by element instead of guessing pixels.
- **`wait_for_change`** blocks until the screen actually changes (for example, a page finishing loading) and returns the new frame, instead of the agent polling with repeated screenshots. It takes `timeout_ms` (default 10000) and `poll_ms` (default 250).
- **`paste`** enters long or Unicode text through the clipboard — faster and more reliable than typing it character by character.
- **`key`** presses a chord such as `ctrl+s`; modifiers are `cmd`, `ctrl`, `alt`, `shift`, and chords may include the `f1`–`f12` function keys.

### The action budget

Each session has a **per-session action budget** (`max_actions`, default `80`). When a session hits the cap it halts and hands control back to the human, and the tool reports `action_budget_exhausted`. This is the backstop against a runaway loop quietly hammering the desktop forever.

### Taking the machine back

You do not have to wait for the budget. `/pause` hands the cursor and keyboard back to you. The computer-use session stays alive and refuses further actions until you `/resume` — unlike `/stop`, which tears the session down. Both are owner-only, and both take effect between the session's actions, so the agent stops at the next step rather than mid-drag.

`/resume` only lifts the block; it does not pick the task back up on its own. Tell the agent what you want next and it continues from there.

## Safety model

Computer use has no filesystem sandbox around it — its `:gui_control` policy class carries no path enforcement, because "click here, type this" is not something a path allowlist can reason about. Its safety comes from three deterministic floors plus one prompt-level principle.

### Access posture, derived from the sandbox mode

There is **no separate computer-use permission knob**. The posture — how far the tool may go — is derived **1:1 from your [sandbox](/docs/sandbox) mode**. An `open` sandbox is an `open` desktop.

| Sandbox mode | Computer-use posture |
|---|---|
| `strict` | **Look only.** Read-only actions run; every mutating action is refused. This is the one deterministic, code-enforced floor. |
| `standard` (default) | Acts freely, but the agent **confirms with the owner before anything irreversible** — delete, send, purchase, sign out, overwrite. This is the agent's own judgment (it can see the screen), applied through its instructions, not a blocking gate. |
| `open` | Acts **autonomously**, including ordinary destructive steps, but still pauses to confirm a **truly catastrophic** action — mass deletion, wiping or formatting data, sending money. A higher bar than `standard`, not zero. |

Only `strict` (refuse-all-mutations) is a hard code gate; the `standard` and `open` "confirm first" behavior is a principle folded into the tool's live instructions each turn. To change the posture, change the sandbox mode — for example `fermix sandbox mode strict` to drop the desktop to look-only.

### Attended-origin gate

A host desktop must have a watching human. A computer-use session therefore starts **only from an attended origin** — an interactive chat message, a `fermix ask` command, or the [voice companion](/docs/realtime-voice), where a person is present to see what happens and stop it. **Scheduled jobs, cron runs, and detached `/background` runs fail closed**: a run with no live chat or call for a human to watch and abort can never begin driving the live session, regardless of the access posture. Access governs *what* the tool does; the origin gate governs *whether an unwatched run may start it at all*.

A session started from a voice call is additionally **bound to that call**: when you end the call, the desktop session is torn down, so it never outlives the moment a human was watching. Over voice, the agent sees the screen the same way it does in chat — each screenshot is delivered to the voice model as an image, and its on-screen text is wrapped as untrusted content just like everywhere else.

### Operator-only, never delegated

The `:gui_control` policy class is **operator-only** — a guest (an allowlisted chat sender who is not the owner) cannot reach it — and it is **never delegated to subagents**. When the main agent fans work out to helper subagents, `:gui_control` is stripped from what they can do, so control of your desktop stays with the one agent you are talking to.

### Screenshots are untrusted

Everything the tool reads from the screen is treated as **external, untrusted content**. A malicious page or dialog on screen is, in effect, an attacker trying to inject instructions. This is real risk *mitigation, not containment*: on-screen text could in principle talk a `standard`-mode agent out of asking before a destructive step. Only `strict` and the attended-origin gate are absolute. Enable computer use with that understanding.

### Sharing the machine (coexistence)

You and the agent have one cursor between you, so the agent steps aside when you are using it. Before any action that would move the pointer or type — clicking, dragging, scrolling, moving the mouse, typing, pressing a key, or pasting — Fermix checks how long the machine has been idle. If you are actively using it, the agent waits a few seconds for a gap, then either takes its turn or holds the action back and tells you it stepped aside.

Note that `mouse_move` counts here even though the table above lists it as read-only: it changes nothing on screen, but it does warp your pointer out from under your hand. "Disturbing you" and "changing something" are different properties.

The check knows the difference between your input and the agent's own. The operating system reports only how long it has been since *any* input, including the agent's synthetic clicks, so Fermix blames recent input on a human only when the agent has itself been quiet — otherwise its own click would look like you and it would stall waiting for itself.

This is a **courtesy, not a floor**. Idle detection is macOS-only. Where it is unavailable the action proceeds, so this is a courtesy that keeps the agent from fighting you for the cursor, not a safety boundary — the hard floors are the `strict` posture and the attended-origin gate above. When you want the machine back outright rather than for a moment, use `/pause`.

Two keys control it: `courtesy` (default `"yield"`; set it to `"off"` to disable stepping aside) and `courtesy_idle_ms` (default `1000` — how much quiet counts as you being idle). How long the agent waits for a gap before giving up its turn is internal and not configurable.

## Sessions and lifecycle

A computer-use session is created **lazily, per conversation**: the OS-driver process opens the first time the tool is actually used in that conversation and is then reused across every action in it — not spun up per turn. The driver runs as a **separate operating-system process, outside the BEAM VM**, so a stuck or misbehaving driver cannot take the daemon down with it. When the session tears down it releases any input it was holding, so a drag or a held key can never be left stuck.

On session start the daemon performs a **version handshake** with the helper and **refuses to run one whose wire protocol does not match** the running build — for example if a partial upgrade left an old binary behind. It fails loud rather than sending commands a mismatched binary might misinterpret. The fix is to **reinstall the helper** from the setup **Plugins** page card so it matches the daemon.

## Configuration reference

The `[fermix_core.computer_use]` section:

```toml
[fermix_core.computer_use]
enabled                  = true    # off by default; the feature flag that turns it on
display                  = 0       # which monitor to drive (0 = primary)
screenshot_after         = true    # return a fresh screenshot after every mutating action
max_actions              = 80      # per-session action budget; the session halts at the cap
max_retained_screenshots = 3       # how many recent screenshots stay in the model's context
courtesy                 = "yield" # step aside for a present human; "off" disables it
courtesy_idle_ms         = 1000    # how much input quiet counts as you being idle
```

`courtesy` is written as a string, since TOML has no atom type. There is no key for how long the agent waits for a gap before stepping aside — that bound is internal.

Note what is **not** here:

- **There is no `access` key.** The posture mirrors `[sandbox] mode` (above); it is deliberately not configurable on its own.
- **There is no `mode` key.** Computer use is host-desktop control only — a leftover `mode` entry from an old config is ignored and cleaned up on the next save.

Changes to this section apply on the next daemon restart.

## Troubleshooting

Most problems surface in `fermix doctor`'s **computer use** line. What it says and what to do:

| Message | Meaning | Fix |
|---|---|---|
| `disabled` | The feature flag is off. | Set `[fermix_core.computer_use] enabled = true` and restart. |
| `enabled but the helper isn't installed — install it from setup` | Flag is on, binary is missing. | Install the helper from the setup **Plugins** page. |
| `screen capture and input control granted (<display server>)` | All good. | — |
| `screen capture OK but input control is NOT granted — clicks and keystrokes are silently dropped` | The classic macOS trap: Accessibility is missing. | System Settings → Privacy & Security → **Accessibility**. |
| `screen capture is NOT granted — captures return wallpaper only` | macOS Screen Recording is missing. | System Settings → Privacy & Security → **Screen Recording**. |
| Wayland input/capture messages | You are on a Wayland session, which is unsupported. | Log in to an **X11** session instead. |

Once doctor can probe the helper, that line also carries the installed helper's version — ` · sidecar compux v<version>` — the one-glance way to confirm which sidecar build you actually have after a bump; the not-installed warning instead names the version an install would fetch.

Other conditions you may hit at runtime:

- **`no build for this machine` on install** — you are on an unsupported target (an Intel Mac, or a non-X11/non-Apple-Silicon host). Computer use cannot run there; there is no workaround.
- **`no capturable display` at action time** — the screen is locked, the display is asleep, or the process has no active GUI session. Computer use cannot see or control the desktop until there is an unlocked, awake display; retrying will not help until that changes.
- **Protocol mismatch on session start** — the installed helper's wire protocol does not match the daemon (usually a partial upgrade). Reinstall the helper so it matches the running build.
- **A permission is reported missing though the "Fermix Computer Use" box looks checked** — a stale grant from an older build is shadowing the signed app's identity. Remove the **Fermix Computer Use** row under **System Settings → Privacy & Security → Screen Recording** (or run `tccutil reset ScreenCapture io.tezra.fermix.computer-use`) and grant again. Do not remove a row named plain **Fermix** to fix a computer-use permission — that one belongs to the FermixPet voice companion. First-time installs are unaffected.

## Related pages

- [Sandbox and permissions](/docs/sandbox) — the mode that determines computer use's access posture, plus the shorter summary of this relationship.
- [Capabilities and tools](/docs/capabilities-and-tools) — the full built-in tool set and the `:gui_control` policy class.
- [Tool reference](/docs/tool-reference) — the `computer_use` tool's parameter table.
- [Configuration](/docs/configuration) — the complete `config.toml` reference.
- [Ingress and trust](/docs/ingress-and-trust) — operator versus guest trust, which gates who can reach `:gui_control`.
- [Plugins](/docs/plugins) — how the native helper is delivered and verified.