# Realtime voice

> How to set up and use FermixPet, the optional macOS voice companion that lets you talk to your Fermix agent out loud — what it is, what it can access, how it works, and how to turn it on.

Realtime voice lets you *talk* to your Fermix agent instead of typing. You open a call from **FermixPet** — a small macOS desktop companion — speak naturally, and the agent answers out loud in real time, running the same tools and drawing on the same memory it uses in chat.

It is **off by default** and **macOS-only**, and you opt in deliberately (it needs an OpenAI key and a daemon restart). It is a shipped feature, not an experiment — but because a live microphone and spoken tool-running are involved, Fermix keeps it behind an explicit switch. This page covers what it is, what it can reach on your machine, how it works, and the three ways to turn it on.

## How it works

FermixPet is a SwiftUI app (macOS 13 Ventura or newer) that attaches to your already-running Fermix daemon (the background service) over a **local Unix-domain socket** — a private channel between two programs on the same computer. The daemon owns everything that matters: your provider key, the prompt, the tools, memory, and the cost caps. The companion only captures your microphone and plays back audio.

```
FermixPet (SwiftUI, macOS 13+)
   ↕ Unix socket  ~/.fermix/realtime.sock  (permission mode 0600)
LocalVoiceSocket  (accept loop, up to 4 connected clients)
   ↕ per-call GenServer
SessionServer    (one per active call)
   ↕ WebSocket
OpenAI Realtime API  (configured model, default gpt-realtime-2)
```

The socket is created by the daemon with permission mode `0600`, so **only the same operating-system user can connect** — that file permission *is* the authentication. Nothing on your network can reach it.

Because FermixPet and the daemon ship on separate release cadences, the two negotiate a protocol version when the app connects: the daemon accepts the current version and the previous one. If the pair drifts too far apart — a much newer FermixPet against an old daemon, say — the connection is refused with a version-mismatch error naming the side that must update, rather than misbehaving. Upgrade the older half.

**The call model is click-to-start, then hands-free.** You click to open a call; from then on FermixPet streams your microphone continuously and OpenAI's server-side voice-activity detection decides when you have started and stopped talking — you do not press a button per sentence. You can interrupt the agent mid-sentence (barge-in) and it stops to listen. You click again to end the call. When no call is open, **no audio leaves your machine.**

There is **no always-listening or wake-word mode** — capture begins only when you open a call. Adding one would require a change to the trust model, because a voice session is treated as *you*, present at the keyboard (see below).

**Spoken replies are deliberately short.** On top of the agent's usual prompt, a call layers `REALTIME.md`, a voice-only rules file seeded into `~/.fermix/bootstrap/<agent_id>/`: lead with the answer, one sentence by default and two at most, speak prose rather than read markdown aloud, and never pre-announce or narrate a tool call — act, then give the result. It is an ordinary file you can edit. Seeding never overwrites a file that already exists, so an install made before a wording change keeps its own copy until you adopt it; `fermix doctor` flags that drift. See [prompt and compaction](/docs/prompt-and-compaction).

## What it can access

A voice call runs at **operator trust** — the same level of access you have as the machine's owner typing in chat. In other words, when you talk to FermixPet you are talking to the full agent, not a stripped-down version.

**It can:**

- Answer questions and hold a conversation, using web search and page fetches for current facts.
- Read and write files, run shell commands, and use git — all passed through the **same [sandbox](/docs/sandbox) as the main agent** (the `[sandbox] mode` and command profile decide what it may touch; there is no separate voice permission).
- Store and recall [memory](/docs/memory), create and manage [scheduled jobs](/docs/scheduled-jobs), run [skills](/docs/skills), and call any connected [plugins](/docs/plugins).

**It cannot:**

- **Send file attachments** or **generate images.** Both deliver their result *through a chat channel*, and a voice call has no channel to deliver into — so those two tools are simply not offered, and the agent will say so if you ask.
- **Delegate to sub-agents.** The `subagents` tool is withheld from voice as well: a fan-out that blocks for minutes while workers grind does not fit a live call.

**Driving your desktop (when computer use is on).** If you have enabled [computer use](/docs/computer-use) — off by default — a voice call *can* control your desktop. Because you are on the call, the session counts as "attended" and is allowed to start, and the agent genuinely sees your screen: each screenshot is delivered to the voice model as an image, not just described. The same safeguards apply as anywhere else — screen contents are treated as untrusted input, your [sandbox](/docs/sandbox) mode sets the access posture (`strict` is look-only), and it is owner-only — plus one specific to voice: the desktop session is bound to the call, so when you hang up it is torn down and can never outlive the conversation.

**To narrow what voice can reach, change the sandbox mode** (for example `fermix sandbox mode strict` to keep it inside the workspace) — the same lever that governs the text agent. Older per-voice restriction knobs (`tool_policy`, `allow_network_tools`) were removed and now raise an error if left in your config; the sandbox is the single control.

**Memory works both ways, carefully scoped.** Voice sees the same distilled profile (`USER.md` / `MEMORY.md`) and the same memory store as your text conversations, so it already knows what you have told Fermix elsewhere. But the raw call transcript is kept in its own thread (channel `realtime`) and is **not** mixed into your text chats — and by default transcripts are not stored at all (see [Transcripts and cost](#transcripts-and-cost)).

## Setting it up

You need one prerequisite and then any one of three ways to flip it on.

### Prerequisite: an OpenAI Realtime key

Realtime voice requires a real **OpenAI Platform API key** (an `sk-…` key with Realtime access and billing). A ChatGPT or Codex OAuth login does **not** authorize it — even if `openai_codex` is your text provider, voice uses the plain OpenAI key independently. The key reuses your OpenAI provider key slot, and setting a key *only* for voice does not make OpenAI your primary text provider.

### 1. Web setup — the Realtime tab (easiest)

Open the setup page (`fermix setup` prints a one-time link) and go to the **Realtime** tab ("Voice companion"). Set the status to **Enabled**, paste your OpenAI key (leave it blank to reuse a key you already stored), and optionally set the model, voice, reasoning effort, max session minutes, max cost per call, and whether to save transcripts. Click **Save**, then **Apply & restart** — the change takes effect after the daemon restarts.

### 2. Terminal wizard

Run `fermix setup` (or `fermix setup --reconfigure` to revisit answers). The wizard has a **Realtime** step that asks whether to enable the voice companion, for your OpenAI key (skipped if it already has one), and then the voice, session-minute cap, cost cap, and whether to save transcripts. The daemon restarts to apply.

### 3. Scripted flags (headless or container installs)

For unattended setup, pass the answers as flags:

```bash
fermix setup --reconfigure \
  --realtime-enabled \
  --realtime-voice marin \
  --realtime-max-session-minutes 15 \
  --realtime-max-cost-cents 100
```

Add `--realtime-api-key sk-...` if the OpenAI key is not already stored, and `--realtime-persist-transcripts` to save transcripts. This writes:

```toml
[fermix_core.realtime]
enabled = true
provider = "openai"
model = "gpt-realtime-2"
reasoning_effort = "low"
voice = "marin"
max_session_minutes = 15
max_estimated_cost_cents_per_session = 100
persist_transcripts = false
```

### After you enable it

The voice socket only exists inside a running daemon that was launched with `fermix run`. **The installed OS service already does this** — it runs `fermix run` under the hood — so on a normal install you just restart the service (`fermix restart`) and the socket opens. If you run the daemon by hand instead, start it with `fermix run` (a bare `fermix start` on its own does not open the socket). Confirm it came up with `fermix voice status` (below), then install and open FermixPet.

## Installing FermixPet

FermixPet lives in its own repository — [`tezra-io/fermix-macos`](https://github.com/tezra-io/fermix-macos) — and ships as a signed, notarized app. Install it with Homebrew:

```bash
brew install --cask tezra-io/tap/fermixpet
open -a FermixPet
```

That puts `FermixPet.app` in `/Applications`. If you would rather not use Homebrew, download the DMG from the [fermix-macos releases page](https://github.com/tezra-io/fermix-macos/releases) and drag the app to Applications.

If you previously ran a self-built copy from `~/Applications`, remove it and reset the microphone grant — otherwise the two copies fight over the permission:

```bash
rm -rf ~/Applications/FermixPet.app
tccutil reset Microphone io.tezra.FermixPet
```

To build from source — or to develop against a local dev daemon instead of the installed service — work from a fermix-macos checkout, where the SwiftPM sources live under `Apps/FermixPet/`. Use the build script rather than `swift run`: it stages a proper `.app` bundle so the Dock icon, Quit, and the microphone-permission prompt behave correctly.

```bash
cd Apps/FermixPet
FERMIX_HOME=$HOME/.fermix-dev ./script/build_and_run.sh
```

The daemon always publishes the socket at `FERMIX_HOME/realtime.sock` — `~/.fermix/realtime.sock` unless you moved the home. The app resolves the path on its own side: `FERMIX_REALTIME_SOCKET` if set, else `FERMIX_HOME/realtime.sock`, else `~/.fermix/realtime.sock`. `FERMIX_REALTIME_SOCKET` is a client-only override the daemon never reads, so it helps only when it points at a path some daemon actually created. A GUI launch — the Homebrew cask, the DMG, the Dock — inherits no shell environment, so neither variable is set and the app always targets `~/.fermix/realtime.sock`. To point a GUI-launched FermixPet at a non-default daemon, attach the environment yourself:

```bash
open --env FERMIX_HOME=$HOME/.fermix-dev -a FermixPet
```

## Checking status

```bash
fermix voice status
fermix voice status --json
```

This reports whether the daemon is online, whether voice is enabled, the provider and model, the socket path, whether a companion is currently connected, and a `realtime key` line showing whether the OpenAI Platform key Realtime needs is present (a Codex OAuth login does not count). `fermix doctor` runs the same check as a `realtime voice` row and warns when voice is enabled without an OpenAI key. The `--json` form is machine-readable, and the `/health/ready` HTTP endpoint includes the same realtime status alongside provider and channel health.

## Transcripts and cost

**Transcripts are off by default.** When `persist_transcripts = true`, the final spoken text of each user and assistant turn is stored as `voice_turn` rows under the `realtime` channel in the same local memory database as your other messages — and those turns feed the normal background [memory review](/docs/memory), so what you say by voice can update your profile. Raw audio is **never** stored (setting `persist_audio = true` is rejected at startup).

**Every call is cost-capped.** Input audio cost is estimated live on each chunk; output audio cost is recorded from OpenAI's usage report when a response finishes. The cap check uses whichever is higher. When a call reaches `max_estimated_cost_cents_per_session` (default 100 cents = $1.00) or `max_session_minutes` (default 15), the session is dropped and FermixPet shows a "limit reached" notice. Every call is also fully traced — see [traces and telemetry](/docs/traces-and-telemetry).

## Configuration reference

### `[fermix_core.realtime]` keys

| Key | Default | Notes |
|-----|---------|-------|
| `enabled` | `false` | Master switch. |
| `provider` | `"openai"` | Only allowed value. |
| `model` | `"gpt-realtime-2"` | One of `gpt-realtime-2.1-mini`, `gpt-realtime-2.1`, `gpt-realtime-2`; any other value fails at startup. Selectable from a dropdown in web setup. |
| `reasoning_effort` | `"low"` | Reasoning effort sent on the Realtime session (`minimal`/`low`/`medium`/`high`/`xhigh`); `low` is OpenAI's recommended starting point for a voice agent. |
| `voice` | `"marin"` | One of the official OpenAI Realtime voices: `marin`, `sage`, `verse`, `cedar`, `alloy`, `ash`, `ballad`, `coral`, `echo`, `shimmer` (setup's dropdown recommends the first four). Any other value fails at startup. |
| `input_audio_format` / `output_audio_format` | `"pcm16"` | PCM16 only. |
| `transcription_model` | `"whisper-1"` | Live partial transcripts of your speech. |
| `max_chunk_bytes` | `16384` | Wire framing chunk size. |
| `max_response_output_tokens` | `4096` | Per-response token cap. |
| `max_session_minutes` | `15` | Hard per-call duration ceiling. |
| `max_estimated_cost_cents_per_session` | `100` | Per-call cost cap, in cents. |
| `persist_transcripts` | `false` | Store spoken text as local memory. |
| `persist_audio` | `false` | Raw-audio persistence; setting `true` fails at startup (not supported). |

Removed keys — `activation`, `turn_detection`, `max_buffer_chunks`, `idle_timeout_ms`, `max_input_audio_seconds_per_session`, `tool_policy`, `allow_network_tools` — cause the daemon to refuse to boot with a message telling you to delete them. Realtime now runs a single full-duplex mode with server-side voice detection, and its tool access mirrors the main agent (tune it through the sandbox, not these knobs).

### Environment variable overrides

| Variable | Description |
|----------|-------------|
| `FERMIX_REALTIME_ENABLED` | Enable the voice companion. |
| `FERMIX_REALTIME_PROVIDER` | Realtime provider; only `openai` is supported. |
| `FERMIX_REALTIME_MODEL` | Override the OpenAI Realtime model. |
| `FERMIX_REALTIME_VOICE` | Override the voice. |
| `FERMIX_REALTIME_MAX_SESSION_MINUTES` | Per-call duration cap. |
| `FERMIX_REALTIME_MAX_COST_CENTS` | Per-call cost cap (short form). |
| `FERMIX_REALTIME_MAX_ESTIMATED_COST_CENTS_PER_SESSION` | Long-form alias for the cost cap. |
| `FERMIX_REALTIME_PERSIST_TRANSCRIPTS` | Persist final transcripts as local memory. |

For precedence rules and secret storage, see [configuration](/docs/configuration) and [auth and secrets](/docs/auth-and-secrets).

## Troubleshooting

| Symptom | Resolution |
|---------|------------|
| `daemon: offline` | Start Fermix: `fermix start` (service) or `fermix run` (manual). |
| `setup_required` | Set `OPENAI_API_KEY` or store an OpenAI key through `fermix setup`. |
| Companion can't reach the mic | Grant microphone access in macOS Privacy & Security — the entry is listed as **Fermix**, the app's display name, not FermixPet. To start the prompt over, reset it: `tccutil reset Microphone io.tezra.FermixPet`. |
| App is quarantined after a manual copy | Remove the Gatekeeper flag: `xattr -dr com.apple.quarantine FermixPet.app`. |
| Voice enabled but nothing connects | Confirm the daemon runs via `fermix run` (the installed service does) and that `fermix voice status` shows the socket alive. |
| Companion refuses to connect after upgrading one side | The two fell outside the shared version window. The error names the direction: `client_too_old` means update FermixPet, `client_too_new` means update Fermix (then `fermix restart`). |

## Development

To run the full daemon with realtime enabled from source, without packaging a binary:

```bash
FERMIX_HOME=$HOME/.fermix-dev \
OPENAI_API_KEY=sk-... \
FERMIX_REALTIME_ENABLED=true \
FERMIX_REALTIME_MODEL=gpt-realtime-2 \
mix fermix.dev
```

Pass `--no-realtime` to `mix fermix.dev` to skip the voice socket when working on other subsystems. The readiness banner confirms which sockets opened.

### Wire protocol and version window

The socket speaks newline-delimited JSON. Every connection opens with a mandatory handshake — the app sends `client_hello` carrying its `protocol_version`, the daemon replies `server_hello` advertising the inclusive `{min_version, max_version}` range it accepts — and no other event is serviced until it completes. That range is an **N/N-1 window**: the current version and the one before it, so an app exactly one version behind still connects.

The window also fixes the rollout order, and **the daemon ships first**. A wire change lands in a daemon release that adds the new version while still accepting the old one; only then does a FermixPet release start speaking it, so a released app never requires a version the released daemon lacks. Rollback runs in reverse.

`FermixCore.Realtime.Protocol` is the source of truth, and the machine-readable export of the contract lives beside it under `apps/fermix_core/priv/realtime/`: `PROTOCOL.md` (transport, handshake state machine, rollout order), `protocol.schema.json`, and golden `fixtures/*.jsonl` for both directions. fermix-macos vendors those pinned by checksum rather than hand-copying the shapes, and a contract test asserts the export never drifts from the module — so if you are writing your own client, that directory is what to read.

See also: [channels](/docs/channels), [configuration](/docs/configuration), [auth and secrets](/docs/auth-and-secrets), [capabilities and tools](/docs/capabilities-and-tools), [sandbox](/docs/sandbox).