Realtime voice

Realtime voice lets you talk to your Fermix agent instead of typing. You open a call from the Fermix macOS app, speak naturally, and the agent answers out loud in real time, running the same tools and drawing on the same memory it uses in chat.

It is off by default and macOS-only, and you opt in deliberately (it needs an OpenAI key and a daemon restart). It is a shipped feature, not an experiment — but because a live microphone and spoken tool-running are involved, Fermix keeps it behind an explicit switch. Enable it from the app’s Voice settings or the standalone setup wizard.

How it works

The Fermix macOS app requires macOS 15 Sequoia or newer and connects its voice companion to the Fermix daemon over a local Unix-domain socket — a private channel between two programs on the same computer. The daemon owns everything that matters: your provider key, the prompt, the tools, memory, and the cost caps. The companion only captures your microphone and plays back audio.

Fermix voice companion (SwiftUI, macOS 15+)
   ↕ Unix socket  ~/.fermix/realtime.sock  (permission mode 0600)
LocalVoiceSocket  (accept loop, up to 4 connected clients)
   ↕ per-call GenServer
SessionServer    (one per active call)
   ↕ WebSocket
OpenAI Realtime API  (configured model, default gpt-realtime-2)

The socket is created by the daemon with permission mode 0600, so only the same operating-system user can connect — that file permission is the authentication. Nothing on your network can reach it.

The app and daemon negotiate a protocol version when the app connects. Because their protocol versions can differ, the daemon accepts the current version and the previous one. If the pair drifts too far apart — a newer app against an old daemon — the connection is refused with a version-mismatch error naming the side that must update, rather than misbehaving. Upgrade the older half.

The call model is click-to-start, then hands-free. You click to open a call; from then on the app streams your microphone continuously and OpenAI’s server-side voice-activity detection decides when you have started and stopped talking — you do not press a button per sentence. You can interrupt the agent mid-sentence (barge-in) and it stops to listen. You click again to end the call. When no call is open, no audio leaves your machine.

There is no always-listening or wake-word mode — capture begins only when you open a call. Adding one would require a change to the trust model, because a voice session is treated as you, present at the keyboard (see below).

Spoken replies are deliberately short. On top of the agent’s usual prompt, a call layers REALTIME.md, a voice-only rules file seeded into ~/.fermix/bootstrap/<agent_id>/: lead with the answer, one sentence by default and two at most, speak prose rather than read markdown aloud, and never pre-announce or narrate a tool call — act, then give the result. Two more of its rules: watch the screen when the task is on it (a single screenshot is one look; the live feed is for anything you follow or do together), and work through a multi-step sequence in silence, speaking when it completes, when it needs something from you, or when it is blocked. It is an ordinary file you can edit — editing it changes how the next call behaves. Seeding never overwrites a file that already exists, so an install made before a wording change keeps its own copy until you adopt it; fermix doctor flags that drift. See prompt and compaction.

What it can access

A voice call runs at operator trust — the same level of access you have as the machine’s owner typing in chat. In other words, when you talk to the voice companion you are talking to the full agent, not a stripped-down version.

It can:

• Answer questions and hold a conversation, using web search and page fetches for current facts.
• Read and write files, run shell commands, and use git — all passed through the same sandbox as the main agent (the [sandbox] mode and command profile decide what it may touch; there is no separate voice permission).
• Store and recall memory, create and manage scheduled jobs, run skills, and call any connected plugins.

It cannot:

• Send file attachments or generate images. Both deliver their result through a chat channel, and a voice call has no channel to deliver into — so those two tools are simply not offered, and the agent will say so if you ask.
• Delegate to sub-agents. The subagents tool is withheld from voice as well: a fan-out that blocks for minutes while workers grind does not fit a live call.
• Launch coding-agent runs or approve directory access. The coding agents run tools and the request_directory_access approval prompt both need a chat surface to deliver a reply or approval into, so neither is offered on a call.

Driving your desktop (when computer use is on). If you have enabled computer use — off by default — a voice call can control your desktop. Because you are on the call, the session counts as “attended” and is allowed to start, and the agent genuinely sees your screen: each screenshot is delivered to the voice model as an image, not just described. The same safeguards apply as anywhere else — screen contents are treated as untrusted input, your sandbox mode sets the access posture (strict is look-only), and it is owner-only — plus one specific to voice: the desktop session is bound to the call, so when you hang up it is torn down and can never outlive the conversation.

Watching your screen during a call (screen sharing). You do not have to ask. When the task you bring to a call concerns your screen — an app or form you have open, a page you are reading or working through together, a game you play together — the agent starts a live feed of that display on its own and says it has started; the task itself is the request. You can still ask it to watch explicitly, and it never claims to see your screen while no share is running — if you describe something it cannot see, it says so and starts watching. Changed frames join the conversation quietly as context: a still screen sends nothing, and a frame never makes the agent speak on its own, so it answers from what is on screen when you ask rather than narrating it. The frames are low-detail awareness images; to read fine print or aim a click the agent takes a fresh computer use screenshot, and acting on what it sees still goes through computer_use or browser with their unchanged safeguards.

Screen sharing rides on computer use, so it needs computer use enabled and installed and the macOS Screen Recording permission granted — the same helper, the same grant. If the permission is missing the call refuses to start watching and says so, rather than streaming blank frames and describing an empty desktop. It is voice-only: a text conversation is never offered it, and takes a one-off screenshot instead. Sharing is never silent, and it is scoped to the call. The agent announces every start — a standing rule in its voice rules file, applied by its own judgment — while the boundaries around it are code-enforced: “stop watching” always ends it, ending the call always ends the feed, and the next call begins with sharing off even though voice mode never went away. A brief provider-connection drop that reconnects mid-call is not a new call — the feed suspends for the reconnect and resumes on its own without re-asking. It also stops itself, telling the agent why, if screen capture keeps failing or the feed reaches its share of the call’s cost cap. To withhold the capability entirely, set screen_share = false under [fermix_core.realtime].

To narrow what voice can reach, change the sandbox mode (for example fermix sandbox mode strict to keep it inside the workspace) — the same lever that governs the text agent. Older per-voice restriction knobs (tool_policy, allow_network_tools) were removed and now raise an error if left in your config; the sandbox is the single control. A strict sandbox watches and narrates but refuses to click.

Memory works both ways, carefully scoped. Voice sees the same distilled profile (USER.md / MEMORY.md) and the same memory store as your text conversations, so it already knows what you have told Fermix elsewhere. But the raw call transcript is kept in its own thread (channel realtime) and is not mixed into your text chats — and by default transcripts are not stored at all (see Transcripts and cost).

Setting it up

You need an OpenAI Realtime key, then enable voice through the app or standalone setup.

Prerequisite: an OpenAI Realtime key

Realtime voice requires a real OpenAI Platform API key (an sk-… key with Realtime access and billing). A ChatGPT or Codex OAuth login does not authorize it — even if openai_codex is your text provider, voice uses the plain OpenAI key independently. The key reuses your OpenAI provider key slot, and setting a key only for voice does not make OpenAI your primary text provider.

In the macOS app

Open Settings → Voice, enable the voice companion, and add an OpenAI API key if one is not stored. Choose the model, voice, reasoning effort, call limits, and transcript setting, then apply the requested restart. Open Pet and choose Begin voice call. macOS asks for microphone access on the first call; allow Fermix. End voice call stops capture.

Web setup on a standalone install

Open the setup page (fermix setup prints a one-time link) and go to the Realtime tab (“Voice companion”). Set the status to Enabled, paste your OpenAI key (leave it blank to reuse a key you already stored), and optionally set the model, voice, reasoning effort, max session minutes, max cost per call, and whether to save transcripts. Click Save, then Apply & restart — the change takes effect after the daemon restarts.

Terminal wizard on a standalone install

Run fermix setup (or fermix setup --reconfigure to revisit answers). The wizard has a Realtime step that asks whether to enable the voice companion, for your OpenAI key (skipped if it already has one), and then the voice, session-minute cap, cost cap, and whether to save transcripts. The daemon restarts to apply.

Scripted flags (standalone installs)

For unattended setup, pass the answers as flags:

fermix setup --reconfigure \
  --realtime-enabled \
  --realtime-voice marin \
  --realtime-max-session-minutes 15 \
  --realtime-max-cost-cents 100

Add --realtime-api-key sk-... if the OpenAI key is not already stored, and --realtime-persist-transcripts to save transcripts. This writes:

[fermix_core.realtime]
enabled = true
provider = "openai"
model = "gpt-realtime-2"
reasoning_effort = "low"
voice = "marin"
max_session_minutes = 15
max_estimated_cost_cents_per_session = 100
persist_transcripts = false

Screen sharing has no setup flag: it is on by default under [fermix_core.realtime] screen_share and inert until computer use is enabled. See configuration.

After you enable it

The voice socket exists only while the daemon is running with realtime enabled. The macOS app starts and restarts its own background service. On a standalone service install, run fermix restart after changing voice settings; for a foreground daemon, use fermix run. Confirm the socket came up with fermix voice status, then open the app’s voice companion.

Installing the companion

The voice companion is included in the Fermix macOS app. Follow that page for installation or migration from a standalone Homebrew daemon; a second voice-only app is not required.

The daemon publishes realtime.sock inside its Fermix home. The app resolves that home from ~/Library/Application Support/Fermix/launcher.json, shared with its background service and bundled CLI. Select an existing home during onboarding or carry it over with fermix migrate-to-app. The app does not read FERMIX_HOME or FERMIX_REALTIME_SOCKET from its environment.

The app sources live in tezra-io/fermix-macos, under Apps/Fermix/. For a local GUI build, use its script/build_and_run.sh rather than swift run; it stages an app bundle with the microphone permission identity. Follow that repository’s development setup to connect an isolated daemon: passing FERMIX_HOME to the GUI no longer selects its home.

Checking status

fermix voice status
fermix voice status --json

This reports whether the daemon is online, whether voice is enabled, the provider and model, the socket path, whether a companion is currently connected, and a realtime key line showing whether the OpenAI Platform key Realtime needs is present (a Codex OAuth login does not count). fermix doctor runs the same check as a realtime voice row and warns when voice is enabled without an OpenAI key. The --json form is machine-readable, and the /health/ready HTTP endpoint includes the same realtime status alongside provider and channel health.

Transcripts and cost

Transcripts are off by default. When persist_transcripts = true, the final spoken text of each user and assistant turn is stored as voice_turn rows under the realtime channel in the same local memory database as your other messages — and those turns feed the normal background memory review, so what you say by voice can update your profile. Raw audio is never stored (setting persist_audio = true is rejected at startup).

Every call is cost-capped. Input audio cost is estimated live on each chunk, and the full spend — audio, text, and images, in and out — is accumulated from OpenAI’s usage report as each response finishes, with each response counted exactly once and cached input billed once at the cached rate. The cap check uses whichever of the two is higher. When a call reaches max_estimated_cost_cents_per_session (default 100 cents = $1.00) or max_session_minutes (default 15), the call ends and the app shows a “limit reached” notice naming the reason. Screen watching is metered separately, per frame actually sent, against its own share — half — of the call’s cost cap, so the agent’s own high-detail screenshots cannot exhaust the watching budget; when the feed reaches its share, the feed stops and the call continues, with the agent told why. Every call is also fully traced — see traces and telemetry.

Configuration reference

[fermix_core.realtime] keys

Key	Default	Notes
enabled	false	Master switch.
provider	"openai"	Only allowed value.
model	"gpt-realtime-2"	One of gpt-realtime-2.1-mini, gpt-realtime-2.1, gpt-realtime-2; any other value fails at startup. Selectable from a dropdown in web setup.
reasoning_effort	"low"	Reasoning effort sent on the Realtime session (minimal/low/medium/high/xhigh); low is OpenAI’s recommended starting point for a voice agent.
voice	"marin"	One of the official OpenAI Realtime voices: marin, sage, verse, cedar, alloy, ash, ballad, coral, echo, shimmer (setup’s dropdown recommends the first four). Any other value fails at startup.
input_audio_format / output_audio_format	"pcm16"	PCM16 only.
transcription_model	"whisper-1"	Live partial transcripts of your speech.
max_chunk_bytes	16384	Wire framing chunk size.
max_response_output_tokens	4096	Per-response token cap.
max_session_minutes	15	Hard per-call duration ceiling.
max_estimated_cost_cents_per_session	100	Per-call cost cap, in cents.
persist_transcripts	false	Store spoken text as local memory.
persist_audio	false	Raw-audio persistence; setting true fails at startup (not supported).

Removed keys — activation, turn_detection, max_buffer_chunks, idle_timeout_ms, max_input_audio_seconds_per_session, tool_policy, allow_network_tools — cause the daemon to refuse to boot with a message telling you to delete them. Realtime now runs a single full-duplex mode with server-side voice detection, and its tool access mirrors the main agent (tune it through the sandbox, not these knobs).

Environment variable overrides

Variable	Description
FERMIX_REALTIME_ENABLED	Enable the voice companion.
FERMIX_REALTIME_PROVIDER	Realtime provider; only openai is supported.
FERMIX_REALTIME_MODEL	Override the OpenAI Realtime model.
FERMIX_REALTIME_VOICE	Override the voice.
FERMIX_REALTIME_MAX_SESSION_MINUTES	Per-call duration cap.
FERMIX_REALTIME_MAX_COST_CENTS	Per-call cost cap (short form).
FERMIX_REALTIME_MAX_ESTIMATED_COST_CENTS_PER_SESSION	Long-form alias for the cost cap.
FERMIX_REALTIME_PERSIST_TRANSCRIPTS	Persist final transcripts as local memory.

For precedence rules and secret storage, see configuration and auth and secrets.

Troubleshooting

A transient provider hiccup does not end a live call. Routine recoverable errors mid-call are logged and traced rather than tearing the call down, and a dropped provider connection triggers a few bounded reconnect attempts while the app shows a reconnecting state (audio arriving during the gap is dropped, and screen sharing resumes afterward). Only genuinely terminal reasons end a call, each with a named reason: you hang up, the cost cap, the session-minute cap, or a provider connection that stays down after the reconnect attempts are exhausted.

Symptom	Resolution
daemon: offline	Enable the background service in the macOS app. For a standalone daemon, use fermix start (service) or fermix run (manual).
setup_required	Set OPENAI_API_KEY or store an OpenAI key through fermix setup.
Companion can’t reach the mic	Grant microphone access in macOS Privacy & Security — the entry is listed as Fermix. To start the prompt over, reset it: tccutil reset Microphone io.tezra.FermixPet.
macOS refuses to open the app	Reinstall the signed, notarized app from the official release.
Voice enabled but nothing connects	Confirm the daemon runs via fermix run (the installed service does) and that fermix voice status shows the socket alive.
Companion refuses to connect after upgrading one side	The two fell outside the shared version window. The error names the direction: client_too_old means update the app, client_too_new means update Fermix (then fermix restart).

Development

To run the full daemon with realtime enabled from source, without packaging a binary:

FERMIX_HOME=$HOME/.fermix-dev \
OPENAI_API_KEY=sk-... \
FERMIX_REALTIME_ENABLED=true \
FERMIX_REALTIME_MODEL=gpt-realtime-2 \
mix fermix.dev

Pass --no-realtime to mix fermix.dev to skip the voice socket when working on other subsystems. The readiness banner confirms which sockets opened.

Wire protocol and version window

The socket speaks newline-delimited JSON. Every connection opens with a mandatory handshake — the app sends client_hello carrying its protocol_version, the daemon replies server_hello advertising the inclusive {min_version, max_version} range it accepts — and no other event is serviced until it completes. That range is an N/N-1 window: the current version and the one before it, so an app exactly one version behind still connects.

The window also fixes the rollout order, and the daemon ships first. A wire change lands in a daemon release that adds the new version while still accepting the old one; only then does an app release start speaking it, so a released app never requires a version the released daemon lacks. Rollback runs in reverse.

FermixCore.Realtime.Protocol is the source of truth, and the machine-readable export of the contract lives beside it under apps/fermix_core/priv/realtime/: PROTOCOL.md (transport, handshake state machine, rollout order), protocol.schema.json, and golden fixtures/*.jsonl for both directions. fermix-macos vendors those pinned by checksum rather than hand-copying the shapes, and a contract test asserts the export never drifts from the module — so if you are writing your own client, that directory is what to read.

See also: channels, configuration, auth and secrets, capabilities and tools, sandbox.