Traces and telemetry

Fermix records what it does, so you can see and debug its activity. Every Fermix component emits structured telemetry (machine-readable records of events as they happen) through a central handler that converts events into JSONL trace files (one JSON record per line) under ~/.fermix/traces/YYYY-MM-DD/<type>.jsonl, rotated daily (a fresh file each day). This page covers the trace types, event vocabulary, log rotation, health endpoint, and the optional Opik exporter.

Trace files

Trace files live under FERMIX_HOME/traces/ (default ~/.fermix/traces/) in a directory per calendar date. Each event type gets its own file:

File	Contents
llm_call.jsonl	Provider adapter calls
tool_exec.jsonl	Capability execution outcomes
agent_event.jsonl	Agent lifecycle, plugin distribution ops, memory reviews
channel_msg.jsonl	Inbound and outbound channel messages
sandbox_event.jsonl	Sandbox (the safety boundary on file and command access) denials — the path and command decisions that were blocked (allowed decisions are not written)
error.jsonl	Captured failures with stack traces

The handler keeps one open file handle per {date, type} pair and rotates on a date change. Writes are best-effort: a failed trace write is logged but never takes down the runtime.

Override the default directory with FERMIX_TRACE_DIR.

Entry shape

Every line is a JSON object. The handler adds ts (ISO 8601 UTC), type, and agent to every entry, then recursively encodes the payload:

{
  "ts": "2026-05-18T23:21:39.477Z",
  "type": "tool_exec",
  "agent": "main",
  "tool": "file_read",
  "success": true,
  "duration_ms": 12,
  "path": "/Users/me/projects/foo/README.md"
}

Every run carries a session_id. The format encodes the run kind: main-<n> for interactive turns, random hex for subagents (helper agents the main agent spins off for a piece of work), cron_<job>_<ts> for scheduled job runs, and session:<n> for realtime voice calls. Subagents also carry a parent_session so the full delegation tree (which agent spun off which) can be reconstructed by correlating on those fields.

Telemetry event vocabulary

Fermix emits telemetry through shared emitters, and a central handler routes most of those events into the JSONL trace files. The table below lists the events that are written to a trace file and which file each one lands in. Tool and provider events are generic — a new tool reuses [:fermix, :tool, :exec] and a new provider reuses [:fermix, :provider, :call], so both appear in traces automatically.

Event	Trace file	What it records
[:fermix, :provider, :call]	llm_call	A provider adapter call — provider, model, duration_ms, tokens (plus auth_mode on the adapters that have one: Anthropic and xAI). Non-chat calls reuse this event with a deliberately empty tokens map, so no token cost is attributed to them: voice-note transcriptions carry purpose: :transcription and a provider that may be a transcription-only vendor such as deepgram, and generate_image calls carry no purpose
[:fermix, :tool, :exec]	tool_exec	A capability execution — tool, success, duration_ms
[:fermix, :memory, :write]	tool_exec	The background memory reviewer’s durable write (recorded as a tool row)
[:fermix, :mcp, :inbound, :call]	tool_exec	A tool call served to a connected MCP (Model Context Protocol, the standard for exposing tools to AI clients) client
[:fermix, :channel, :message]	channel_msg	An inbound or outbound channel message — direction (:inbound or :outbound)
[:fermix, :sandbox, :decision]	sandbox_event	A denied file or command decision — capability, policy_class, reason_tag, resource (allowed decisions are not written)
[:fermix, :agent, :message]	agent_event	A completed MainAgent turn — iterations, total_tokens, duration_ms
[:fermix, :agent, :message_error]	agent_event	A turn that failed
[:fermix, :agent, :start|:stop|:task_start|:task_complete]	agent_event	Agent-process and subagent-task lifecycle markers
[:fermix, :agent, :prompt_context|:history], [:fermix, :channel, :reply], [:fermix, :capabilities, :select]	agent_event	Turn-prep and delivery markers (prompt assembly, history load, reply sent, capability set chosen)
[:fermix, :skill, :invoke|:journal_write]	agent_event	A skill run and its journal write
[:fermix, :job, :run_start|:run_complete|:run_error]	agent_event	A scheduled job run’s lifecycle (carries job_id and run_id)
[:fermix, :memory, :review]	agent_event	The background memory review (“dreaming”) run closer
[:fermix, :plugin, :dist]	agent_event	A plugin install, uninstall, or garbage-collection op
[:fermix, :tool_search, :query]	agent_event	A tool-schema deferral bridge search (match_count; a miss is the health signal)
[:fermix, :provider, :failover]	agent_event	A provider failover on a transient error
[:fermix, :timeout, :expired]	agent_event	A fired failure deadline (FermixCore.Timeouts) — the timeout name and elapsed ms, correlated by session_id
[:fermix, :channel, :stream]	agent_event	Streaming-turn bookends (stream_started / stream_finalized; interim draft edits are telemetry-only and not written)
[:fermix, :mcp, :inbound, :tools_listed]	agent_event	An inbound MCP tools/list served to a client
[:fermix, :soul_curation, :run_start|:run_complete|:run_error]	agent_event	The /soul review bounded draft call lifecycle
[:fermix, :realtime, :call_start|session_created|session_updated|provider_error|reconnect|call_stop]	agent_event	The realtime voice call lifecycle

Some telemetry the runtime emits is not written to the JSONL files — it is available only to a live telemetry subscriber (attach your own handler). These include auto-compaction ([:fermix, :compaction, :auto] and :auto_skipped, where older messages are summarized to stay within the model’s context limit), slash-command dispatch and rejection ([:fermix, :command, :received] and :unauthorized), agent-loop iterations ([:fermix, :agent, :iteration]), unrecognized-model sightings ([:fermix, :model_catalog, :unknown_model]), attachment-send failures ([:fermix, :channel, :media_send_error]), and a MainAgent-unavailable signal seen during a restart ([:fermix, :dispatcher, :agent_unavailable]).

Content capture

By default, prompt bodies, LLM responses, and tool inputs/outputs are omitted from traces entirely (records note that an event happened but do not store its content, and stay small). Setting FERMIX_TRACE_CONTENT=1 enables full-fidelity capture: input/output bodies are written whole with no truncation, and a failed browser action additionally carries the profile’s recent console/JS-exception buffer (the browser’s recent log and error messages) in its error details. The Opik exporter (see below) enables full content capture automatically when it starts; set FERMIX_TRACE_CONTENT=0 to suppress that even when Opik is on.

Log file

Standard runtime logs go to ~/.fermix/logs/fermix.log (override with FERMIX_LOG_FILE). Rotation is configured at 10 MB per file, keeping up to 5 files (when a file fills up, Fermix starts a new one and discards the oldest). Every line is redacted before it is written: a formatter wraps both the file handler and the console handler, so all log output — OTP crash reports included, which no source-level redaction can see — is scanned for credential-shaped tokens and each match replaced with a [REDACTED:<vendor>] marker. See auth and secrets for the vendor patterns covered. Tail the log (stream new lines as they are written) with:

fermix logs -f

Inspecting traces

# List today's trace files
ls ~/.fermix/traces/$(date +%F)/

# Filter shell tool calls
jq 'select(.tool == "shell")' ~/.fermix/traces/$(date +%F)/tool_exec.jsonl

# Find slow LLM calls
jq 'select(.duration_ms > 5000)' ~/.fermix/traces/$(date +%F)/llm_call.jsonl

# Show errors
jq '.' ~/.fermix/traces/$(date +%F)/error.jsonl

Traces from multiple days can be queried by globbing across date directories:

jq 'select(.success == false)' ~/.fermix/traces/*/tool_exec.jsonl

Health endpoint

GET /health/ready returns a structured JSON report covering:

• Config file path and FERMIX_HOME
• Provider status (per configured provider)
• Per-channel status (Telegram, WhatsApp, Slack, Discord, Signal)
• Memory backend process status
• Realtime voice status

GET /health/live returns process liveness only (no dependency checks). /health is a backward-compatible alias for /health/ready.

curl -s http://127.0.0.1:4030/health/ready | jq .

See configuration for the bind address (FERMIX_HTTP_BIND) and channels for per-channel setup.

Opik exporter

fermix_opik is a built-in component that exports Fermix telemetry to an Opik instance (Opik is an external dashboard for inspecting LLM traces and costs). It is compiled into every build but is entirely inert unless enabled: it does nothing and adds zero overhead when off.

Enable it by setting FERMIX_OPIK_ENABLED=1 in the daemon (the long-running background Fermix process) environment:

FERMIX_OPIK_ENABLED=1 fermix run

When running as a system service, a shell export alone never reaches the daemon process. Set the variable and then reinstall the unit (the OS service definition: launchd on macOS, systemd on Linux) so it is captured into that definition:

# After exporting FERMIX_OPIK_ENABLED=1 in your shell
fermix service install

fermix doctor reports whether the exporter is off, enabled-but-not-bundled, or ready (with the resolved endpoint and project name).

When enabled, the exporter:

• Reads the shared provider, tool, job, and lifecycle telemetry correlated by session_id and parent_session.
• Maps each run (interactive turn, subagent, scheduled job, memory review) into Opik’s trace model, nesting child runs under their parent.
• Renders LLM spans (the per-call records in Opik’s view) with provider, model, and token usage for Opik’s cost attribution. A new provider must register a short identifier for cost to work.
• Enables full content capture by default (prompts and responses are passed whole to Opik spans). Override with FERMIX_TRACE_CONTENT=0 to suppress.

The exporter reads events the runtime emits regardless. Its absence or failure never affects a turn or reply.

Environment variable	Default	Description
FERMIX_OPIK_ENABLED	unset (off)	Set to 1 to activate the exporter
FERMIX_OPIK_BASE_URL	http://localhost:5173/api (local Opik)	Override the Opik endpoint URL
FERMIX_OPIK_PROJECT	fermix	Override the Opik project name
FERMIX_TRACE_CONTENT	unset	1 forces full content capture; 0 suppresses it even when Opik is enabled; absent defaults to off unless Opik is enabled
FERMIX_TRACE_DIR	~/.fermix/traces	Override the JSONL trace output directory
FERMIX_LOG_FILE	~/.fermix/logs/fermix.log	Override the rotating log file path

For webhook configuration that feeds into the observability pipeline, see webhooks. For all configuration keys and environment variables, see configuration.