Computer use

Computer use lets the agent operate your actual desktop — moving the real mouse, typing on the real keyboard, and reading the screen from screenshots — one action at a time. It exists for the last-resort case: a graphical app with no API, no command line, and no way in except the way a person would use it. It is experimental and off by default, and it is the most dangerous thing Fermix can do: it acts on your live, logged-in session, there is no undo, and the screenshots it reads are treated as untrusted content (a page could try to talk the agent into doing something). Turn it on only when you want the agent to drive the actual machine.

This is host-desktop control, not web automation. For clicking around a website or a JavaScript-rendered page, the separate browser tool drives a managed Chrome/Chromium process and is the right tool — see capabilities and tools. Computer use is for the desktop itself: native apps, system dialogs, anything outside a browser tab. There is no “browser mode” here.

How it works

The capability is a single tool, computer_use, that performs one action per call. The loop is always the same: take a screenshot to see the current state, then act on what you saw — click, type, press a key, scroll, or drag — using pixel coordinates read from that screenshot. Every action that changes something returns a fresh screenshot so the agent can check the result and retry a missed click, rather than acting blind. Because each step is verified against a new picture of the screen, the agent self-corrects as it goes.

Supported platforms

Computer use ships a native helper binary — compux, an open-source Rust sidecar — and that binary exists only for two targets:

Platform	Supported
Apple Silicon macOS (M-series)	Yes
Linux x86_64, X11 session	Yes
Intel Macs	No — install fails loudly
Linux on Wayland	No — capture and input injection are blocked; use an X11 session
Everything else	No — install fails with a clear “no build for this machine” error

There is no silent degradation: an unsupported machine refuses the install with a typed error naming the target, rather than half-working.

Enabling it

Turning computer use on is one operator step — set the feature flag in ~/.fermix/config.toml and restart:

[fermix_core.computer_use]
enabled = true

The change applies on the next daemon restart (fermix restart). On a real daemon boot, before the readiness checks run, Fermix downloads the native helper matching that build if it is not already present. The helper is a small standalone binary that drives the desktop — it registers no tools of its own. It is fetched from the compux project’s own signed release and verified against a SHA-256 checksum baked into your Fermix build, then cached under ~/.fermix/plugins/compux/. fermix plugins does not manage it. (This one component is delivered as a checksum-verified download rather than through the signed plugin-tarball pipeline the other catalog plugins use — see plugins.)

That automatic fetch is deliberately narrow: it happens on daemon boot only — not during fermix setup — it is capped at about 30 seconds so a slow network cannot hold up the daemon, and it fails soft. If the download does not land, computer use stays off and you get it back on the next restart or by enabling it from the setup card. It exists so that an upgrade which moves the pinned helper version does not leave computer use silently dead until you notice.

You can also do it from the setup Plugins page: open the Computer Use card and enable it there, and Fermix writes the flag and fetches the helper for you.

The computer_use tool becomes visible to the model only once both hold — the feature is enabled and the helper is installed. A daemon that has one but not the other never advertises the tool at all, so the model cannot try to use a capability that is not actually ready.

On the setup Plugins page, once the helper is installed its card shows the feature’s own readiness — Ready when it is enabled and installed, or Needs setup otherwise — instead of a generic plugin enable/disable state.

Operating-system permissions

Driving the desktop needs the operating system’s blessing, and this is deliberately a diagnostic, not part of the enable gate. The reason is a specific, nasty failure mode:

On macOS, taking screenshots needs Screen Recording permission, but moving the mouse and typing needs Accessibility permission — and they are granted separately. If Screen Recording is granted but Accessibility is not, the agent can see the screen perfectly while every click and keystroke is silently dropped. It looks like the tool can only look and never act.

Because screenshots keep working without the input grant, a missing permission must inform the operator rather than hide the tool — which is why permission state is not folded into the “is the tool ready?” check. To see where you stand, run:

fermix doctor

and read its computer use line. It probes the helper without triggering a permission prompt and reports Screen Recording and Accessibility distinctly, with the exact fix. On macOS the fix it names for the silent-click trap is System Settings → Privacy & Security → Accessibility; for missing capture it names Screen Recording in the same pane. On Linux it reports whether you are on X11 (supported) or Wayland (not).

On macOS the helper runs as its own Developer-ID-signed, notarized Fermix.app — a permanent bundle identity of its own, rather than one inherited from the daemon build — so a Screen Recording or Accessibility grant sticks across upgrades and appears as a single Fermix Computer Use entry in System Settings, instead of re-prompting and leaving a new row behind every time you upgrade. The plain Fermix row you may also see in those panes is a different app — the FermixPet voice companion. The two share an icon on purpose, so only the name tells them apart. On the setup Doctor page, running the probe reports the same permission state; there, when a grant is actually missing on macOS, it also offers a Grant macOS permissions button that raises the Screen Recording and Accessibility prompts — registering the app with the OS so it shows up in System Settings — up front, rather than surprising you on the agent’s first screenshot.

The action loop

computer_use takes an action plus whatever arguments that action needs (coordinates, text, a key chord). Actions split into read-only actions, which never change the screen and always run, and mutating actions, which do change it:

Kind	Actions
Read-only (always allowed)	screenshot, mouse_move, wait, wait_for_change, inspect, elements
Mutating	left_click, right_click, double_click, left_click_drag, scroll, type, paste, key

A few of these are worth calling out:

• screenshot with a region zooms. Pass a rectangle {x, y, w, h} in the current screenshot’s pixel space and you get back a magnified crop of just that area. Then pass the same region on the follow-up click or drag and give coordinates read from the magnified image — Fermix maps them back to the real pixel. This makes clicks on small or dense targets land far more accurately than guessing at full-screen coordinates.
• inspect (read-only, macOS) reports the accessibility element under a point — its role and label, e.g. “a button titled Delete” — so the agent can confirm it is about to click the right control before a consequential action. It informs the agent’s own judgment; it is not a blocking gate.
• elements (read-only, macOS) lists the interactive elements on screen, each with a click point, so the agent targets by element instead of guessing pixels.
• wait_for_change blocks until the screen actually changes (for example, a page finishing loading) and returns the new frame, instead of the agent polling with repeated screenshots. It takes timeout_ms (default 10000) and poll_ms (default 250).
• paste enters long or Unicode text through the clipboard — faster and more reliable than typing it character by character.
• key presses a chord such as ctrl+s; modifiers are cmd, ctrl, alt, shift, and chords may include the f1–f12 function keys.

The action budget

Each session has a per-session action budget (max_actions, default 80). When a session hits the cap it halts and hands control back to the human, and the tool reports action_budget_exhausted. This is the backstop against a runaway loop quietly hammering the desktop forever.

Taking the machine back

You do not have to wait for the budget. /pause hands the cursor and keyboard back to you. The computer-use session stays alive and refuses further actions until you /resume — unlike /stop, which tears the session down. Both are owner-only, and both take effect between the session’s actions, so the agent stops at the next step rather than mid-drag.

/resume only lifts the block; it does not pick the task back up on its own. Tell the agent what you want next and it continues from there.

Safety model

Computer use has no filesystem sandbox around it — its :gui_control policy class carries no path enforcement, because “click here, type this” is not something a path allowlist can reason about. Its safety comes from three deterministic floors plus one prompt-level principle.

Access posture, derived from the sandbox mode

There is no separate computer-use permission knob. The posture — how far the tool may go — is derived 1:1 from your sandbox mode. An open sandbox is an open desktop.

Sandbox mode	Computer-use posture
strict	Look only. Read-only actions run; every mutating action is refused. This is the one deterministic, code-enforced floor.
standard (default)	Acts freely, but the agent confirms with the owner before anything irreversible — delete, send, purchase, sign out, overwrite. This is the agent’s own judgment (it can see the screen), applied through its instructions, not a blocking gate.
open	Acts autonomously, including ordinary destructive steps, but still pauses to confirm a truly catastrophic action — mass deletion, wiping or formatting data, sending money. A higher bar than standard, not zero.

Only strict (refuse-all-mutations) is a hard code gate; the standard and open “confirm first” behavior is a principle folded into the tool’s live instructions each turn. To change the posture, change the sandbox mode — for example fermix sandbox mode strict to drop the desktop to look-only.

Attended-origin gate

A host desktop must have a watching human. A computer-use session therefore starts only from an attended origin — an interactive chat message, a fermix ask command, or the voice companion, where a person is present to see what happens and stop it. Scheduled jobs, cron runs, and detached /background runs fail closed: a run with no live chat or call for a human to watch and abort can never begin driving the live session, regardless of the access posture. Access governs what the tool does; the origin gate governs whether an unwatched run may start it at all.

A session started from a voice call is additionally bound to that call: when you end the call, the desktop session is torn down, so it never outlives the moment a human was watching. Over voice, the agent sees the screen the same way it does in chat — each screenshot is delivered to the voice model as an image, and its on-screen text is wrapped as untrusted content just like everywhere else.

Operator-only, never delegated

The :gui_control policy class is operator-only — a guest (an allowlisted chat sender who is not the owner) cannot reach it — and it is never delegated to subagents. When the main agent fans work out to helper subagents, :gui_control is stripped from what they can do, so control of your desktop stays with the one agent you are talking to.

Screenshots are untrusted

Everything the tool reads from the screen is treated as external, untrusted content. A malicious page or dialog on screen is, in effect, an attacker trying to inject instructions. This is real risk mitigation, not containment: on-screen text could in principle talk a standard-mode agent out of asking before a destructive step. Only strict and the attended-origin gate are absolute. Enable computer use with that understanding.

Sharing the machine (coexistence)

You and the agent have one cursor between you, so the agent steps aside when you are using it. Before any action that would move the pointer or type — clicking, dragging, scrolling, moving the mouse, typing, pressing a key, or pasting — Fermix checks how long the machine has been idle. If you are actively using it, the agent waits a few seconds for a gap, then either takes its turn or holds the action back and tells you it stepped aside.

Note that mouse_move counts here even though the table above lists it as read-only: it changes nothing on screen, but it does warp your pointer out from under your hand. “Disturbing you” and “changing something” are different properties.

The check knows the difference between your input and the agent’s own. The operating system reports only how long it has been since any input, including the agent’s synthetic clicks, so Fermix blames recent input on a human only when the agent has itself been quiet — otherwise its own click would look like you and it would stall waiting for itself.

This is a courtesy, not a floor. Idle detection is macOS-only. Where it is unavailable the action proceeds, so this is a courtesy that keeps the agent from fighting you for the cursor, not a safety boundary — the hard floors are the strict posture and the attended-origin gate above. When you want the machine back outright rather than for a moment, use /pause.

Two keys control it: courtesy (default "yield"; set it to "off" to disable stepping aside) and courtesy_idle_ms (default 1000 — how much quiet counts as you being idle). How long the agent waits for a gap before giving up its turn is internal and not configurable.

Sessions and lifecycle

A computer-use session is created lazily, per conversation: the OS-driver process opens the first time the tool is actually used in that conversation and is then reused across every action in it — not spun up per turn. The driver runs as a separate operating-system process, outside the BEAM VM, so a stuck or misbehaving driver cannot take the daemon down with it. When the session tears down it releases any input it was holding, so a drag or a held key can never be left stuck.

On session start the daemon performs a version handshake with the helper and refuses to run one whose wire protocol does not match the running build — for example if a partial upgrade left an old binary behind. It fails loud rather than sending commands a mismatched binary might misinterpret. The fix is to reinstall the helper from the setup Plugins page card so it matches the daemon.

Configuration reference

The [fermix_core.computer_use] section:

[fermix_core.computer_use]
enabled                  = true    # off by default; the feature flag that turns it on
display                  = 0       # which monitor to drive (0 = primary)
screenshot_after         = true    # return a fresh screenshot after every mutating action
max_actions              = 80      # per-session action budget; the session halts at the cap
max_retained_screenshots = 3       # how many recent screenshots stay in the model's context
courtesy                 = "yield" # step aside for a present human; "off" disables it
courtesy_idle_ms         = 1000    # how much input quiet counts as you being idle

courtesy is written as a string, since TOML has no atom type. There is no key for how long the agent waits for a gap before stepping aside — that bound is internal.

Note what is not here:

• There is no access key. The posture mirrors [sandbox] mode (above); it is deliberately not configurable on its own.
• There is no mode key. Computer use is host-desktop control only — a leftover mode entry from an old config is ignored and cleaned up on the next save.

Changes to this section apply on the next daemon restart.

Troubleshooting

Most problems surface in fermix doctor’s computer use line. What it says and what to do:

Message	Meaning	Fix
disabled	The feature flag is off.	Set [fermix_core.computer_use] enabled = true and restart.
enabled but the helper isn't installed — install it from setup	Flag is on, binary is missing.	Install the helper from the setup Plugins page.
screen capture and input control granted (<display server>)	All good.	—
screen capture OK but input control is NOT granted — clicks and keystrokes are silently dropped	The classic macOS trap: Accessibility is missing.	System Settings → Privacy & Security → Accessibility.
screen capture is NOT granted — captures return wallpaper only	macOS Screen Recording is missing.	System Settings → Privacy & Security → Screen Recording.
Wayland input/capture messages	You are on a Wayland session, which is unsupported.	Log in to an X11 session instead.

Once doctor can probe the helper, that line also carries the installed helper’s version — · sidecar compux v<version> — the one-glance way to confirm which sidecar build you actually have after a bump; the not-installed warning instead names the version an install would fetch.

Other conditions you may hit at runtime:

• no build for this machine on install — you are on an unsupported target (an Intel Mac, or a non-X11/non-Apple-Silicon host). Computer use cannot run there; there is no workaround.
• no capturable display at action time — the screen is locked, the display is asleep, or the process has no active GUI session. Computer use cannot see or control the desktop until there is an unlocked, awake display; retrying will not help until that changes.
• Protocol mismatch on session start — the installed helper’s wire protocol does not match the daemon (usually a partial upgrade). Reinstall the helper so it matches the running build.
• A permission is reported missing though the “Fermix Computer Use” box looks checked — a stale grant from an older build is shadowing the signed app’s identity. Remove the Fermix Computer Use row under System Settings → Privacy & Security → Screen Recording (or run tccutil reset ScreenCapture io.tezra.fermix.computer-use) and grant again. Do not remove a row named plain Fermix to fix a computer-use permission — that one belongs to the FermixPet voice companion. First-time installs are unaffected.

Related pages

• Sandbox and permissions — the mode that determines computer use’s access posture, plus the shorter summary of this relationship.
• Capabilities and tools — the full built-in tool set and the :gui_control policy class.
• Tool reference — the computer_use tool’s parameter table.
• Configuration — the complete config.toml reference.
• Ingress and trust — operator versus guest trust, which gates who can reach :gui_control.
• Plugins — how the native helper is delivered and verified.