# Traces and telemetry

> How Fermix emits structured JSONL traces, what telemetry events exist, how logs rotate, what /health/ready reports, and how to enable the optional Opik exporter.

Fermix records what it does, so you can see and debug its activity. Every Fermix component emits structured telemetry (machine-readable records of events as they happen) through a central handler that converts events into JSONL trace files (one JSON record per line) under `~/.fermix/traces/YYYY-MM-DD/<type>.jsonl`, rotated daily (a fresh file each day). This page covers the trace types, event vocabulary, log rotation, health endpoint, and the optional Opik exporter.

## Trace files

Trace files live under `FERMIX_HOME/traces/` (default `~/.fermix/traces/`) in a directory per calendar date. Each event type gets its own file:

| File | Contents |
|------|----------|
| `llm_call.jsonl` | Provider adapter calls |
| `tool_exec.jsonl` | Capability execution outcomes |
| `agent_event.jsonl` | Agent lifecycle, plugin distribution ops, memory reviews |
| `channel_msg.jsonl` | Inbound and outbound channel messages |
| `sandbox_event.jsonl` | Sandbox (the safety boundary on file and command access) denials — the path and command decisions that were blocked (allowed decisions are not written) |
| `error.jsonl` | Captured failures with stack traces |

The handler keeps one open file handle per `{date, type}` pair and rotates on a date change. Writes are best-effort: a failed trace write is logged but never takes down the runtime.

Override the default directory with `FERMIX_TRACE_DIR`.

## Entry shape

Every line is a JSON object. The handler adds `ts` (ISO 8601 UTC), `type`, and `agent` to every entry, then recursively encodes the payload:

```json
{
  "ts": "2026-05-18T23:21:39.477Z",
  "type": "tool_exec",
  "agent": "main",
  "tool": "file_read",
  "success": true,
  "duration_ms": 12,
  "path": "/Users/me/projects/foo/README.md"
}
```

Every run carries a `session_id`. The format encodes the run kind: `main-<n>` for interactive turns, random hex for subagents (helper agents the main agent spins off for a piece of work), `cron_<job>_<ts>` for scheduled job runs, and `session:<n>` for realtime voice calls. Subagents also carry a `parent_session` so the full delegation tree (which agent spun off which) can be reconstructed by correlating on those fields.

## Telemetry event vocabulary

Fermix emits telemetry through shared emitters, and a central handler routes most of those events into the JSONL trace files. The table below lists the events that are written to a trace file and which file each one lands in. Tool and provider events are generic — a new tool reuses `[:fermix, :tool, :exec]` and a new provider reuses `[:fermix, :provider, :call]`, so both appear in traces automatically.

| Event | Trace file | What it records |
|-------|-----------|-----------------|
| `[:fermix, :provider, :call]` | `llm_call` | A provider adapter call — `provider`, `model`, `duration_ms`, `tokens` (plus `auth_mode` on the adapters that have one: Anthropic and xAI). Non-chat calls reuse this event with a deliberately empty `tokens` map, so no token cost is attributed to them: voice-note transcriptions carry `purpose: :transcription` and a `provider` that may be a transcription-only vendor such as `deepgram`, and `generate_image` calls carry no `purpose` |
| `[:fermix, :tool, :exec]` | `tool_exec` | A capability execution — `tool`, `success`, `duration_ms` |
| `[:fermix, :memory, :write]` | `tool_exec` | The background memory reviewer's durable write (recorded as a tool row) |
| `[:fermix, :mcp, :inbound, :call]` | `tool_exec` | A tool call served to a connected MCP (Model Context Protocol, the standard for exposing tools to AI clients) client |
| `[:fermix, :channel, :message]` | `channel_msg` | An inbound or outbound channel message — `direction` (`:inbound` or `:outbound`) |
| `[:fermix, :sandbox, :decision]` | `sandbox_event` | A **denied** file or command decision — `capability`, `policy_class`, `reason_tag`, `resource` (allowed decisions are not written) |
| `[:fermix, :agent, :message]` | `agent_event` | A completed MainAgent turn — `iterations`, `total_tokens`, `duration_ms` |
| `[:fermix, :agent, :message_error]` | `agent_event` | A turn that failed |
| `[:fermix, :agent, :start\|:stop\|:task_start\|:task_complete]` | `agent_event` | Agent-process and subagent-task lifecycle markers |
| `[:fermix, :agent, :prompt_context\|:history]`, `[:fermix, :channel, :reply]`, `[:fermix, :capabilities, :select]` | `agent_event` | Turn-prep and delivery markers (prompt assembly, history load, reply sent, capability set chosen) |
| `[:fermix, :skill, :invoke\|:journal_write]` | `agent_event` | A skill run and its journal write |
| `[:fermix, :job, :run_start\|:run_complete\|:run_error]` | `agent_event` | A scheduled job run's lifecycle (carries `job_id` and `run_id`) |
| `[:fermix, :memory, :review]` | `agent_event` | The background memory review ("dreaming") run closer |
| `[:fermix, :plugin, :dist]` | `agent_event` | A plugin install, uninstall, or garbage-collection op |
| `[:fermix, :tool_search, :query]` | `agent_event` | A tool-schema deferral bridge search (`match_count`; a miss is the health signal) |
| `[:fermix, :provider, :failover]` | `agent_event` | A provider failover on a transient error |
| `[:fermix, :timeout, :expired]` | `agent_event` | A fired failure deadline (`FermixCore.Timeouts`) — the timeout `name` and elapsed `ms`, correlated by `session_id` |
| `[:fermix, :channel, :stream]` | `agent_event` | Streaming-turn bookends (`stream_started` / `stream_finalized`; interim draft edits are telemetry-only and not written) |
| `[:fermix, :mcp, :inbound, :tools_listed]` | `agent_event` | An inbound MCP `tools/list` served to a client |
| `[:fermix, :soul_curation, :run_start\|:run_complete\|:run_error]` | `agent_event` | The `/soul review` bounded draft call lifecycle |
| `[:fermix, :realtime, :call_start\|session_created\|session_updated\|provider_error\|reconnect\|call_stop]` | `agent_event` | The realtime voice call lifecycle |

Some telemetry the runtime emits is **not** written to the JSONL files — it is available only to a live telemetry subscriber (attach your own handler). These include auto-compaction (`[:fermix, :compaction, :auto]` and `:auto_skipped`, where older messages are summarized to stay within the model's context limit), slash-command dispatch and rejection (`[:fermix, :command, :received]` and `:unauthorized`), agent-loop iterations (`[:fermix, :agent, :iteration]`), unrecognized-model sightings (`[:fermix, :model_catalog, :unknown_model]`), attachment-send failures (`[:fermix, :channel, :media_send_error]`), and a MainAgent-unavailable signal seen during a restart (`[:fermix, :dispatcher, :agent_unavailable]`).

## Content capture

By default, prompt bodies, LLM responses, and tool inputs/outputs are omitted from traces entirely (records note that an event happened but do not store its content, and stay small). Setting `FERMIX_TRACE_CONTENT=1` enables full-fidelity capture: input/output bodies are written whole with no truncation, and a failed browser action additionally carries the profile's recent console/JS-exception buffer (the browser's recent log and error messages) in its error details. The Opik exporter (see below) enables full content capture automatically when it starts; set `FERMIX_TRACE_CONTENT=0` to suppress that even when Opik is on.

## Log file

Standard runtime logs go to `~/.fermix/logs/fermix.log` (override with `FERMIX_LOG_FILE`). Rotation is configured at 10 MB per file, keeping up to 5 files (when a file fills up, Fermix starts a new one and discards the oldest). Every line is redacted before it is written: a formatter wraps both the file handler and the console handler, so all log output — OTP crash reports included, which no source-level redaction can see — is scanned for credential-shaped tokens and each match replaced with a `[REDACTED:<vendor>]` marker. See [auth and secrets](/docs/auth-and-secrets#log-redaction) for the vendor patterns covered. Tail the log (stream new lines as they are written) with:

```bash
fermix logs -f
```

## Inspecting traces

```bash
# List today's trace files
ls ~/.fermix/traces/$(date +%F)/

# Filter shell tool calls
jq 'select(.tool == "shell")' ~/.fermix/traces/$(date +%F)/tool_exec.jsonl

# Find slow LLM calls
jq 'select(.duration_ms > 5000)' ~/.fermix/traces/$(date +%F)/llm_call.jsonl

# Show errors
jq '.' ~/.fermix/traces/$(date +%F)/error.jsonl
```

Traces from multiple days can be queried by globbing across date directories:

```bash
jq 'select(.success == false)' ~/.fermix/traces/*/tool_exec.jsonl
```

## Health endpoint

`GET /health/ready` returns a structured JSON report covering:

- Config file path and FERMIX_HOME
- Provider status (per configured provider)
- Per-channel status (Telegram, WhatsApp, Slack, Discord, Signal)
- Memory backend process status
- Realtime voice status

`GET /health/live` returns process liveness only (no dependency checks). `/health` is a backward-compatible alias for `/health/ready`.

```bash
curl -s http://127.0.0.1:4030/health/ready | jq .
```

See [configuration](/docs/configuration) for the bind address (`FERMIX_HTTP_BIND`) and [channels](/docs/channels) for per-channel setup.

## Opik exporter

`fermix_opik` is a built-in component that exports Fermix telemetry to an [Opik](https://www.comet.com/opik) instance (Opik is an external dashboard for inspecting LLM traces and costs). It is compiled into every build but is entirely inert unless enabled: it does nothing and adds zero overhead when off.

Enable it by setting `FERMIX_OPIK_ENABLED=1` in the daemon (the long-running background Fermix process) environment:

```bash
FERMIX_OPIK_ENABLED=1 fermix run
```

When running as a system service, a shell export alone never reaches the daemon process. Set the variable and then reinstall the unit (the OS service definition: launchd on macOS, systemd on Linux) so it is captured into that definition:

```bash
# After exporting FERMIX_OPIK_ENABLED=1 in your shell
fermix service install
```

`fermix doctor` reports whether the exporter is off, enabled-but-not-bundled, or ready (with the resolved endpoint and project name).

When enabled, the exporter:

- Reads the shared provider, tool, job, and lifecycle telemetry correlated by `session_id` and `parent_session`.
- Maps each run (interactive turn, subagent, scheduled job, memory review) into Opik's trace model, nesting child runs under their parent.
- Renders LLM spans (the per-call records in Opik's view) with provider, model, and token usage for Opik's cost attribution. A new provider must register a short identifier for cost to work.
- Enables full content capture by default (prompts and responses are passed whole to Opik spans). Override with `FERMIX_TRACE_CONTENT=0` to suppress.

The exporter reads events the runtime emits regardless. Its absence or failure never affects a turn or reply.

| Environment variable | Default | Description |
|---------------------|---------|-------------|
| `FERMIX_OPIK_ENABLED` | unset (off) | Set to `1` to activate the exporter |
| `FERMIX_OPIK_BASE_URL` | `http://localhost:5173/api` (local Opik) | Override the Opik endpoint URL |
| `FERMIX_OPIK_PROJECT` | `fermix` | Override the Opik project name |
| `FERMIX_TRACE_CONTENT` | unset | `1` forces full content capture; `0` suppresses it even when Opik is enabled; absent defaults to off unless Opik is enabled |
| `FERMIX_TRACE_DIR` | `~/.fermix/traces` | Override the JSONL trace output directory |
| `FERMIX_LOG_FILE` | `~/.fermix/logs/fermix.log` | Override the rotating log file path |

For webhook configuration that feeds into the observability pipeline, see [webhooks](/docs/webhooks). For all configuration keys and environment variables, see [configuration](/docs/configuration).