Docs/Concepts/Supervision tree

Supervision tree

How Fermix structures its OTP supervision tree across the core, channels, web, and realtime applications.

This page explains how Fermix keeps its internal moving parts alive and restarts them cleanly when something fails. Fermix runs as a single program built on the BEAM (the Erlang/Elixir virtual machine) using OTP, Elixir’s framework for running and supervising long-lived processes. A supervisor is a process whose only job is to start its child processes and restart them if they crash, and a supervision tree is the nested arrangement of those supervisors and children. Every long-lived piece of state is a supervised process. The application has one root supervisor per sub-app; processes that depend on each other appear in the same ordered child list so that failures and restarts cascade correctly.

For a broader view of how these processes fit together, see architecture and the agent loop.

Core supervisor

Module: FermixCore.Application starts FermixCore.Supervisor. Strategy: :rest_for_one

With :rest_for_one, if a child crashes, every child that was started after it is also stopped and restarted in order. This matters here because most processes in the list depend on earlier ones: the tool registries depend on the ETS tables (fast in-memory tables built into the runtime) owned by Capabilities.Registry, the agent depends on all of the above, and so on. Silently keeping a stale registry alive after a crash would cause subtler failures than a clean restart.

The trade-off: a MainAgent crash also brings down the jobs runner and anything started after it. Any in-flight scheduled work is lost. For a single-user local daemon this is the correct choice over silently broken cross-references.

# Child Purpose
1 FermixCore.CommandHost.Supervisor Dynamic supervisor for per-command CommandHost owners. Every external OS command the daemon runs gets a supervised host that owns the command’s process group and sweeps every descendant when the command ends: a normal exit, requester death, or daemon shutdown sweeps with an immediate group SIGKILL; a wall-clock timeout or an output-cap breach sends a graceful SIGTERM first, drains briefly, then unconditionally SIGKILLs — so a hung skill or shell command can never leak orphaned subprocesses. Children are :temporary (a crashed command is a single-command fault, never restarted); it sits first in the tree so every process that can run a command starts after it exists.
2 Task.Supervisor (FermixCore.TaskSupervisor) Hosts every short-lived background task: per-message agent loops, deferred deliveries, extraction passes.
3 Finch (FermixCore.Finch) Shared outbound HTTP connection pool for provider and channel API calls. Configured here with two pool processes per host and a 15-second idle cap on kept-alive connections, so a request right after the host wakes from sleep is served fresh instead of queuing behind a stale socket being torn down.
4 FermixCore.Trace JSONL trace writer; keeps one open file per {date, type} pair.
5 FermixCore.Auth.TokenSupervisor Always started. Supervises per-OAuth-profile token managers via an internal dynamic supervisor and registry.
6 FermixCore.Auth.TokenManager (conditional) Started when Codex is routable (either as the configured primary or as any fallback in the provider chain); refreshes Codex OAuth tokens.
7 FermixCore.Browser.Supervisor Supervises the browser session registry, dynamic supervisor, and profile manager for the browser tool.
8 FermixCore.Capabilities.Registry Owns the ETS table of all registered capabilities.
9 FermixCore.Capabilities.BuiltinSeeder Restart: :transient. Registers every built-in tool synchronously at boot, then returns :ignore.
10 FermixCore.Sandbox.CommandCapabilities Registers sandbox preset commands as built-in capabilities; depends on the registry being alive.
11 FermixCore.Plugins.Dist.Installer GenServer (a single supervised process that handles one thing at a time) that sweeps transient-staging plugin artifacts at boot and serializes plugin install/uninstall/gc mutations.
12 FermixCore.Plugins.CapabilitySeeder Restart: :temporary. Registers capabilities from any enabled plugins into the capability registry.
13 FermixCore.Agents.SkillRegistry Discovers core, local, and plugin skills and mirrors them into the capability registry.
14 FermixCore.Capabilities.MCP.Supervisor Starts every configured outbound MCP server (an external tool server speaking the Model Context Protocol), discovers its tools, and registers them as :mcp capabilities.
15 FermixCore.Memory.Repo SQLite-backed durable memory store.
16 FermixCore.Memory.ConversationStore ETS hot path for conversation history with SQLite write-through.
17 FermixCore.Memory.Store Key-value memory store backed by ETS and SQLite.
18 FermixCore.Setup.BootReport Aggregates the “daemon is ready” signal consumed by the UI and CLI.
19 FermixCore.Agents.AgentSupervisor Dynamic supervisor for subagent processes (short-lived helper agents the main agent spawns to do a piece of work): the workers behind the subagents tool and delegated skill runs (skill_run).
20 FermixCore.Agents.MainAgent Restart: :permanent. The single persistent main agent.
21 FermixCore.Jobs.RunnerSupervisor Dynamic supervisor for scheduled-job runners.
22 FermixCore.Jobs.Scheduler Claims due jobs and spawns runners. Wakes precisely when the next job is due, with a 60-second reconciliation sweep as a safety net.
23 Fermix.CLI.Daemon (conditional) Unix-socket daemon RPC. Started when :daemon_socket_enabled is set (the fermix run path).
24 FermixCore.Realtime.Supervisor (conditional) Started when realtime voice is enabled in config with the openai provider and an OpenAI API key is present (on the daemon path where the realtime socket is enabled).
25 FermixCore.ComputerUse.Supervisor (conditional) Started when computer use is enabled and the native computer-use helper (a separate program that drives the host desktop) is installed. OS permission state is deliberately not part of this gate — a missing Screen Recording or Accessibility grant is surfaced as a diagnostic by fermix doctor instead of hiding the feature. Supervises per-conversation session processes. Experimental and off by default.

MainAgent and AgentSupervisor

MainAgent (child 20) is the single permanent top-level agent. It owns runtime-context cache state and checks out turn-state snapshots for Gateway.Queue, which enforces one FIFO turn per conversation (first-in, first-out: one message at a time, in arrival order).

AgentSupervisor (child 19) is a dynamic supervisor that starts AgentServer workers on demand. Each AgentServer runs one delegated task with its own agent definition, session ID, provider, registry, and parent metadata. These are the worker processes behind the subagents tool and delegated skill runs (skill_run).

Skill and capability registration order

Children 8 through 14 form a strict dependency chain: the registry must be alive before the seeder populates it, and the seeders must complete before the skill registry and MCP supervisor add their own entries. The :rest_for_one strategy ensures any failure in this range restarts the whole chain in order.

Channels supervisor

Module: FermixChannels.Application Strategy: :one_for_one (each child is restarted on its own if it crashes, without disturbing the others)

# Child Started when
1 FermixChannels.Gateway.Queue Always. FIFO turn queue: enforces one active turn per conversation key.
2 FermixChannels.Gateway.BackgroundSupervisor Always. Dynamic supervisor for background (non-blocking) channel tasks.
3 FermixChannels.Gateway.Commands.Sandbox.Confirmations Always. Tracks pending sandbox grant confirmations from channel slash commands.
4 FermixChannels.Gateway.Commands.Soul.Confirmations Always. Tracks pending soul-command confirmations.
5 FermixChannels.Gateway.Idempotency Always. Idempotency cache for webhooks and outbound media: remembers what was already handled so a retried delivery is not processed twice.
6 FermixChannels.Gateway.AlbumBuffer (Telegram) Always. Coalesces multi-image Telegram media groups arriving across poll cycles into a single turn.
7 FermixChannels.Gateway.AlbumBuffer (WhatsApp) Always. Coalesces per-image WhatsApp webhook deliveries with no media_group_id into a single turn.
8+ Transport children (from ChannelRegistry) Per-channel: Telegram poller, Discord gateway, Signal listener, etc. Started only when the respective channel is enabled with the matching transport mode.

Both album buffers are always started; the channel’s configuration only tunes their debounce window, and a buffer for an unused channel simply sits idle.

WhatsApp and Slack adapters receive inbound messages through Phoenix webhook endpoints rather than long-running supervised processes; they do not appear in the transport children list.

Before starting its children, the channels application fails fast on a misconfigured slash-command registry (a duplicate command name or alias would silently shadow a command, so boot aborts instead). It also refuses to start any remote channel adapter that is enabled but has no owner_user_id or allowed_*_ids configured: the refusal is logged as an error, and the channel stays offline rather than accepting messages from unauthorized senders. See ingress and trust for how sender authorization works.

Web supervisor

Module: FermixWeb.Application Strategy: :one_for_one

  1. FermixWebWeb.Telemetry
  2. DNSCluster
  3. Phoenix.PubSub (named FermixWeb.PubSub)
  4. FermixWebWeb.Endpoint

The endpoint only binds a network socket when server: true is set. This flag is applied at runtime inside the fermix run dispatch path. Other CLI paths (setup, version, and similar) halt before the web application starts.

Realtime supervisor

Module: FermixCore.Realtime.Supervisor Strategy: :one_for_one (conditional child of the core supervisor)

  1. Realtime.SessionSupervisor: dynamic supervisor for per-session servers.
  2. Task.Supervisor: for transient realtime work.
  3. Realtime.LocalVoiceSocket: Unix-domain accept loop.

Boot gating: this supervisor is started only when realtime is enabled in config, the provider is "openai", and an OpenAI API key is present. Otherwise it stays unstarted and adds no overhead.

See realtime voice for configuration details.

Boot sequence

The core application runs the following steps before starting the supervision tree:

FermixCore.Application.start/2

  if Burrito standalone:  cli_dispatch(argv)
  else:                   start_supervision_tree()

start_supervision_tree:
  remember_launch_cwd()                  # used by sandbox path policy
  ConfigStore.ensure_workspace()         # mkdir ~/.fermix/*
  BootstrapRename.run()                  # migrate legacy bootstrap file names
  IdentityName.reconcile()               # rewrite IDENTITY.md Name: to configured assistant name
  AuthStore.validate_permissions!()      # auth.json must be 0600 (or missing)
  setup_file_logger()                    # rotating fermix.log
  redact_default_logger()                # secret-redacting formatter on console/crash logs
  Trace.TelemetryHandler.attach()        # bind :fermix.* telemetry
  Sandbox.DecisionTelemetry.attach()     # bind :fermix.sandbox.decision
  ensure_computer_use_sidecar()          # daemon boot only; downloads a missing native helper
  Supervisor.start_link(children, ...)

AuthStore.validate_permissions!() aborts startup if auth.json exists with permissions other than 0600. This runs before any network process starts so no credential is reachable over a misconfigured socket.

ensure_computer_use_sidecar() runs only on a real daemon boot, and a normal boot does no network work at all: it returns immediately when computer use is disabled or the matching native helper is already installed. The one case it acts on is a daemon that has computer use enabled but no helper matching the build it just started — the state an upgrade lands in — and then it downloads the helper once, before the gates in child 25 and the tool registry read it. The download is bounded at 30 seconds and fail-soft: a slow, failed, or crashing fetch is logged and leaves computer use off, retried on the next daemon start or from the setup card, and never blocks or crashes boot.

  • Architecture: the full umbrella structure and cross-cutting invariants.
  • The agent loop: what MainAgent and AgentLoop do inside a turn.
  • Traces and telemetry: what FermixCore.Trace writes and how telemetry handlers attach.
  • Scheduled jobs: how Jobs.Scheduler and Jobs.RunnerSupervisor work.
  • Channels: which adapters are supervised and which use webhook ingress.
  • MCP: how Capabilities.MCP.Supervisor discovers and registers outbound MCP tools.
  • Realtime voice: gating conditions and session lifecycle for the realtime supervisor.

Next steps