Realtime voice

Realtime voice lets you talk to your Fermix agent instead of typing. You open a call from FermixPet — a small macOS desktop companion — speak naturally, and the agent answers out loud in real time, running the same tools and drawing on the same memory it uses in chat.

It is off by default and macOS-only, and you opt in deliberately (it needs an OpenAI key and a daemon restart). It is a shipped feature, not an experiment — but because a live microphone and spoken tool-running are involved, Fermix keeps it behind an explicit switch. This page covers what it is, what it can reach on your machine, how it works, and the three ways to turn it on.

How it works

FermixPet is a SwiftUI app (macOS 13 Ventura or newer) that attaches to your already-running Fermix daemon (the background service) over a local Unix-domain socket — a private channel between two programs on the same computer. The daemon owns everything that matters: your provider key, the prompt, the tools, memory, and the cost caps. The companion only captures your microphone and plays back audio.

FermixPet (SwiftUI, macOS 13+)
   ↕ Unix socket  ~/.fermix/realtime.sock  (permission mode 0600)
LocalVoiceSocket  (accept loop, up to 4 connected clients)
   ↕ per-call GenServer
SessionServer    (one per active call)
   ↕ WebSocket
OpenAI Realtime API  (configured model, default gpt-realtime-2)

The socket is created by the daemon with permission mode 0600, so only the same operating-system user can connect — that file permission is the authentication. Nothing on your network can reach it.

Because FermixPet and the daemon ship on separate release cadences, the two negotiate a protocol version when the app connects: the daemon accepts the current version and the previous one. If the pair drifts too far apart — a much newer FermixPet against an old daemon, say — the connection is refused with a version-mismatch error naming the side that must update, rather than misbehaving. Upgrade the older half.

The call model is click-to-start, then hands-free. You click to open a call; from then on FermixPet streams your microphone continuously and OpenAI’s server-side voice-activity detection decides when you have started and stopped talking — you do not press a button per sentence. You can interrupt the agent mid-sentence (barge-in) and it stops to listen. You click again to end the call. When no call is open, no audio leaves your machine.

There is no always-listening or wake-word mode — capture begins only when you open a call. Adding one would require a change to the trust model, because a voice session is treated as you, present at the keyboard (see below).

Spoken replies are deliberately short. On top of the agent’s usual prompt, a call layers REALTIME.md, a voice-only rules file seeded into ~/.fermix/bootstrap/<agent_id>/: lead with the answer, one sentence by default and two at most, speak prose rather than read markdown aloud, and never pre-announce or narrate a tool call — act, then give the result. It is an ordinary file you can edit. Seeding never overwrites a file that already exists, so an install made before a wording change keeps its own copy until you adopt it; fermix doctor flags that drift. See prompt and compaction.

What it can access

A voice call runs at operator trust — the same level of access you have as the machine’s owner typing in chat. In other words, when you talk to FermixPet you are talking to the full agent, not a stripped-down version.

It can:

• Answer questions and hold a conversation, using web search and page fetches for current facts.
• Read and write files, run shell commands, and use git — all passed through the same sandbox as the main agent (the [sandbox] mode and command profile decide what it may touch; there is no separate voice permission).
• Store and recall memory, create and manage scheduled jobs, run skills, and call any connected plugins.

It cannot:

• Send file attachments or generate images. Both deliver their result through a chat channel, and a voice call has no channel to deliver into — so those two tools are simply not offered, and the agent will say so if you ask.
• Delegate to sub-agents. The subagents tool is withheld from voice as well: a fan-out that blocks for minutes while workers grind does not fit a live call.

Driving your desktop (when computer use is on). If you have enabled computer use — off by default — a voice call can control your desktop. Because you are on the call, the session counts as “attended” and is allowed to start, and the agent genuinely sees your screen: each screenshot is delivered to the voice model as an image, not just described. The same safeguards apply as anywhere else — screen contents are treated as untrusted input, your sandbox mode sets the access posture (strict is look-only), and it is owner-only — plus one specific to voice: the desktop session is bound to the call, so when you hang up it is torn down and can never outlive the conversation.

To narrow what voice can reach, change the sandbox mode (for example fermix sandbox mode strict to keep it inside the workspace) — the same lever that governs the text agent. Older per-voice restriction knobs (tool_policy, allow_network_tools) were removed and now raise an error if left in your config; the sandbox is the single control.

Memory works both ways, carefully scoped. Voice sees the same distilled profile (USER.md / MEMORY.md) and the same memory store as your text conversations, so it already knows what you have told Fermix elsewhere. But the raw call transcript is kept in its own thread (channel realtime) and is not mixed into your text chats — and by default transcripts are not stored at all (see Transcripts and cost).

Setting it up

You need one prerequisite and then any one of three ways to flip it on.

Prerequisite: an OpenAI Realtime key

Realtime voice requires a real OpenAI Platform API key (an sk-… key with Realtime access and billing). A ChatGPT or Codex OAuth login does not authorize it — even if openai_codex is your text provider, voice uses the plain OpenAI key independently. The key reuses your OpenAI provider key slot, and setting a key only for voice does not make OpenAI your primary text provider.

1. Web setup — the Realtime tab (easiest)

Open the setup page (fermix setup prints a one-time link) and go to the Realtime tab (“Voice companion”). Set the status to Enabled, paste your OpenAI key (leave it blank to reuse a key you already stored), and optionally set the model, voice, reasoning effort, max session minutes, max cost per call, and whether to save transcripts. Click Save, then Apply & restart — the change takes effect after the daemon restarts.

2. Terminal wizard

Run fermix setup (or fermix setup --reconfigure to revisit answers). The wizard has a Realtime step that asks whether to enable the voice companion, for your OpenAI key (skipped if it already has one), and then the voice, session-minute cap, cost cap, and whether to save transcripts. The daemon restarts to apply.

3. Scripted flags (headless or container installs)

For unattended setup, pass the answers as flags:

fermix setup --reconfigure \
  --realtime-enabled \
  --realtime-voice marin \
  --realtime-max-session-minutes 15 \
  --realtime-max-cost-cents 100

Add --realtime-api-key sk-... if the OpenAI key is not already stored, and --realtime-persist-transcripts to save transcripts. This writes:

[fermix_core.realtime]
enabled = true
provider = "openai"
model = "gpt-realtime-2"
reasoning_effort = "low"
voice = "marin"
max_session_minutes = 15
max_estimated_cost_cents_per_session = 100
persist_transcripts = false

After you enable it

The voice socket only exists inside a running daemon that was launched with fermix run. The installed OS service already does this — it runs fermix run under the hood — so on a normal install you just restart the service (fermix restart) and the socket opens. If you run the daemon by hand instead, start it with fermix run (a bare fermix start on its own does not open the socket). Confirm it came up with fermix voice status (below), then install and open FermixPet.

Installing FermixPet

FermixPet lives in its own repository — tezra-io/fermix-macos — and ships as a signed, notarized app. Install it with Homebrew:

brew install --cask tezra-io/tap/fermixpet
open -a FermixPet

That puts FermixPet.app in /Applications. If you would rather not use Homebrew, download the DMG from the fermix-macos releases page and drag the app to Applications.

If you previously ran a self-built copy from ~/Applications, remove it and reset the microphone grant — otherwise the two copies fight over the permission:

rm -rf ~/Applications/FermixPet.app
tccutil reset Microphone io.tezra.FermixPet

To build from source — or to develop against a local dev daemon instead of the installed service — work from a fermix-macos checkout, where the SwiftPM sources live under Apps/FermixPet/. Use the build script rather than swift run: it stages a proper .app bundle so the Dock icon, Quit, and the microphone-permission prompt behave correctly.

cd Apps/FermixPet
FERMIX_HOME=$HOME/.fermix-dev ./script/build_and_run.sh

The daemon always publishes the socket at FERMIX_HOME/realtime.sock — ~/.fermix/realtime.sock unless you moved the home. The app resolves the path on its own side: FERMIX_REALTIME_SOCKET if set, else FERMIX_HOME/realtime.sock, else ~/.fermix/realtime.sock. FERMIX_REALTIME_SOCKET is a client-only override the daemon never reads, so it helps only when it points at a path some daemon actually created. A GUI launch — the Homebrew cask, the DMG, the Dock — inherits no shell environment, so neither variable is set and the app always targets ~/.fermix/realtime.sock. To point a GUI-launched FermixPet at a non-default daemon, attach the environment yourself:

open --env FERMIX_HOME=$HOME/.fermix-dev -a FermixPet

Checking status

fermix voice status
fermix voice status --json

This reports whether the daemon is online, whether voice is enabled, the provider and model, the socket path, whether a companion is currently connected, and a realtime key line showing whether the OpenAI Platform key Realtime needs is present (a Codex OAuth login does not count). fermix doctor runs the same check as a realtime voice row and warns when voice is enabled without an OpenAI key. The --json form is machine-readable, and the /health/ready HTTP endpoint includes the same realtime status alongside provider and channel health.

Transcripts and cost

Transcripts are off by default. When persist_transcripts = true, the final spoken text of each user and assistant turn is stored as voice_turn rows under the realtime channel in the same local memory database as your other messages — and those turns feed the normal background memory review, so what you say by voice can update your profile. Raw audio is never stored (setting persist_audio = true is rejected at startup).

Every call is cost-capped. Input audio cost is estimated live on each chunk; output audio cost is recorded from OpenAI’s usage report when a response finishes. The cap check uses whichever is higher. When a call reaches max_estimated_cost_cents_per_session (default 100 cents = $1.00) or max_session_minutes (default 15), the session is dropped and FermixPet shows a “limit reached” notice. Every call is also fully traced — see traces and telemetry.

Configuration reference

[fermix_core.realtime] keys

Key	Default	Notes
enabled	false	Master switch.
provider	"openai"	Only allowed value.
model	"gpt-realtime-2"	One of gpt-realtime-2.1-mini, gpt-realtime-2.1, gpt-realtime-2; any other value fails at startup. Selectable from a dropdown in web setup.
reasoning_effort	"low"	Reasoning effort sent on the Realtime session (minimal/low/medium/high/xhigh); low is OpenAI’s recommended starting point for a voice agent.
voice	"marin"	One of the official OpenAI Realtime voices: marin, sage, verse, cedar, alloy, ash, ballad, coral, echo, shimmer (setup’s dropdown recommends the first four). Any other value fails at startup.
input_audio_format / output_audio_format	"pcm16"	PCM16 only.
transcription_model	"whisper-1"	Live partial transcripts of your speech.
max_chunk_bytes	16384	Wire framing chunk size.
max_response_output_tokens	4096	Per-response token cap.
max_session_minutes	15	Hard per-call duration ceiling.
max_estimated_cost_cents_per_session	100	Per-call cost cap, in cents.
persist_transcripts	false	Store spoken text as local memory.
persist_audio	false	Raw-audio persistence; setting true fails at startup (not supported).

Removed keys — activation, turn_detection, max_buffer_chunks, idle_timeout_ms, max_input_audio_seconds_per_session, tool_policy, allow_network_tools — cause the daemon to refuse to boot with a message telling you to delete them. Realtime now runs a single full-duplex mode with server-side voice detection, and its tool access mirrors the main agent (tune it through the sandbox, not these knobs).

Environment variable overrides

Variable	Description
FERMIX_REALTIME_ENABLED	Enable the voice companion.
FERMIX_REALTIME_PROVIDER	Realtime provider; only openai is supported.
FERMIX_REALTIME_MODEL	Override the OpenAI Realtime model.
FERMIX_REALTIME_VOICE	Override the voice.
FERMIX_REALTIME_MAX_SESSION_MINUTES	Per-call duration cap.
FERMIX_REALTIME_MAX_COST_CENTS	Per-call cost cap (short form).
FERMIX_REALTIME_MAX_ESTIMATED_COST_CENTS_PER_SESSION	Long-form alias for the cost cap.
FERMIX_REALTIME_PERSIST_TRANSCRIPTS	Persist final transcripts as local memory.

For precedence rules and secret storage, see configuration and auth and secrets.

Troubleshooting

Symptom	Resolution
daemon: offline	Start Fermix: fermix start (service) or fermix run (manual).
setup_required	Set OPENAI_API_KEY or store an OpenAI key through fermix setup.
Companion can’t reach the mic	Grant microphone access in macOS Privacy & Security — the entry is listed as Fermix, the app’s display name, not FermixPet. To start the prompt over, reset it: tccutil reset Microphone io.tezra.FermixPet.
App is quarantined after a manual copy	Remove the Gatekeeper flag: xattr -dr com.apple.quarantine FermixPet.app.
Voice enabled but nothing connects	Confirm the daemon runs via fermix run (the installed service does) and that fermix voice status shows the socket alive.
Companion refuses to connect after upgrading one side	The two fell outside the shared version window. The error names the direction: client_too_old means update FermixPet, client_too_new means update Fermix (then fermix restart).

Development

To run the full daemon with realtime enabled from source, without packaging a binary:

FERMIX_HOME=$HOME/.fermix-dev \
OPENAI_API_KEY=sk-... \
FERMIX_REALTIME_ENABLED=true \
FERMIX_REALTIME_MODEL=gpt-realtime-2 \
mix fermix.dev

Pass --no-realtime to mix fermix.dev to skip the voice socket when working on other subsystems. The readiness banner confirms which sockets opened.

Wire protocol and version window

The socket speaks newline-delimited JSON. Every connection opens with a mandatory handshake — the app sends client_hello carrying its protocol_version, the daemon replies server_hello advertising the inclusive {min_version, max_version} range it accepts — and no other event is serviced until it completes. That range is an N/N-1 window: the current version and the one before it, so an app exactly one version behind still connects.

The window also fixes the rollout order, and the daemon ships first. A wire change lands in a daemon release that adds the new version while still accepting the old one; only then does a FermixPet release start speaking it, so a released app never requires a version the released daemon lacks. Rollback runs in reverse.

FermixCore.Realtime.Protocol is the source of truth, and the machine-readable export of the contract lives beside it under apps/fermix_core/priv/realtime/: PROTOCOL.md (transport, handshake state machine, rollout order), protocol.schema.json, and golden fixtures/*.jsonl for both directions. fermix-macos vendors those pinned by checksum rather than hand-copying the shapes, and a contract test asserts the export never drifts from the module — so if you are writing your own client, that directory is what to read.

See also: channels, configuration, auth and secrets, capabilities and tools, sandbox.