Capabilities and tools

Tools are the actions the agent can take: reading a file, running a shell command, searching the web, and so on. Every tool the model can call (whether it is built into Fermix, registered by an installed plugin, or discovered from an MCP server, a standard way for outside programs to expose tools to an AI agent) is one entry in a single shared list, the Capabilities.Registry. There is no separate set of “main agent tools” vs “voice tools” vs “scheduled-job tools.” Trust level and policy filters decide which subset each call site sees. Skills are the one exception: they are instruction packages, not provider-visible tools, and surface through a compact catalog in the prompt plus the skill_view and skill_run built-ins — see skills.

Capability shape

Each registered capability is a struct with these fields:

%Capability{
  name: "file_read",
  description: "Read a UTF-8 text file from the workspace.",
  parameters: %{           # JSON Schema object
    type: "object",
    required: ["path"],
    properties: %{
      path: %{type: "string", description: "..."}
    }
  },
  kind: :builtin,          # :builtin | :skill | :mcp
  executor: {FermixCore.Tools.FileRead, :execute, []},
  hidden_from_agent?: false,
  policy_class: :read_only,
  metadata: %{
    category: :file,
    when_to_use: "...",
    examples: [...],
    failure_modes: [...]
  }
}

The parameters map is a JSON Schema object. AgentLoop passes the formatted schema to the configured provider verbatim. The executor tuple is called with the parsed argument map when the model issues a tool call.

Three kinds of capability

Kind	Source	Registered by
:builtin	Pure-Elixir tool modules compiled into Fermix, sandbox preset commands, and the tools of ready plugins.	BuiltinSeeder at boot; Sandbox.CommandCapabilities for preset shell commands; Plugins.Capabilities for plugin tools (tagged plugin_owned? in metadata; a plugin that runs a local MCP process contributes its tools as :mcp instead).
:skill	Reserved in the schema for skills, but the live runtime deliberately does not register one capability per skill — skills surface through the prompt’s skill catalog and the skill_view / skill_run built-ins.	SkillRegistry keeps skills in its own store and clears any stale :skill entries from the capability registry on reload.
:mcp	Outbound MCP server tools/list response, registered as mcp_<server>_<tool> (a plugin-owned MCP server uses <plugin>_<tool>).	The per-server Capabilities.MCP.Server process when its client connects; each of its tools is unregistered when the connection goes down.

unregister_kind/3 accepts a metadata match map so a whole family of capabilities can be removed atomically — for example, every plugin_owned? tool when plugins reload, or every sandbox preset command when the sandbox config changes.

For more on skills, see skills. For plugins, see plugins. For MCP integration, see MCP.

Policy classes

Every capability carries one policy_class that describes its effect category:

Class	Meaning	Example built-ins
:read_only	Reads data without modifying anything.	file_read, content_search, glob_search, git_read, memory_recall, tool_help, list_jobs, memory_sources_list, send_attachment
:read_write	Writes or mutates local state.	file_write, file_edit, git_write, memory_store, schedule_job, pause_job, resume_job, remove_job, skill_create, model_routing_config
:exec	Spawns an OS process or delegated run. shell goes through the sandbox.	shell, skill_view, skill_run
:network	Outbound HTTP, validated by Net.Guard.	web_fetch, web_search, browser
:external_api	Calls a third-party API, typically an LLM provider. All discovered MCP tools default to this class (a per-tool policy_class override exists in [mcp.servers.<server>.tools.<tool>]).	subagents, generate_image, every :mcp capability
:gui_control	Drives the host desktop by screenshot and mouse/keyboard. Operator-only; never delegated to subagents (helper agents the main agent spins up for a piece of work).	computer_use (experimental, off by default)

Subagent workers always run with the parent trust level’s classes minus :read_write and :gui_control: delegated work can read, browse, and call APIs, but never writes local state or touches the desktop.

Trust model

Trust is stamped at ingress (the point where a message enters Fermix, for example a chat channel or the CLI) and flows through MainAgent to AgentLoop to CapabilityRegistry.list_for/2, which applies the default policy filter for that trust level.

Trust	Default allow	Default deny	Where it is set
:operator	[:read_only, :read_write, :exec, :network, :external_api, :gui_control]	[]	The human owner via CLI, daemon socket (the local connection to the long-running Fermix background process), voice click-to-talk, or the channel’s owner_user_id. Also applies to scheduled jobs created by the operator and bundled or operator-installed skills.
:guest	[:read_only]	[:read_write, :exec, :network, :external_api, :gui_control]	Non-owner senders in allowed_user_ids; plugin-shipped skills (under ~/.fermix/plugins/) whose code the operator has not reviewed.
nil	(treated as :guest)		Any call path that did not set :trust. A missing trust value degrades to read-only; it never silently grants full access.

A list/2 call with no :trust, :policy, or :policy_classes option bypasses the default filter and returns every registered capability. Internal callers such as fermix capabilities use this primitive path.

See ingress and trust for how trust is assigned per channel and per sender.

Filter parameters

CapabilityRegistry.list_for/1,2 accepts these options:

filter() :: [
  allowed_tools:       [String.t()] | nil,
  policy:              policy_spec(),
  policy_classes:      [policy_class()] | nil,
  trust:               :operator | :guest | nil,
  kind:                :builtin | :skill | :mcp | :all,
  excluded_categories: [atom()] | nil,
  excluded_names:      [String.t()] | nil,
  include_hidden?:     boolean()
]

Filters compose in this order: kind, then policy, then allowlist (the explicit list of tool names permitted), then name exclusion, then category exclusion, then hidden filter.

Key semantics:

• allowed_tools: [] means deny all. allowed_tools: nil means no narrowing by name.
• policy: [:read_only] is shorthand for [allow: [:read_only], deny: []].
• excluded_categories: [:scheduling] removes every tool whose metadata.category matches, without listing each tool name.
• include_hidden?: true surfaces capabilities flagged hidden_from_agent?: true. Default is false. Operators use this flag to keep an MCP tool configured but invisible to the model.

Registry API

Call	Notes
register/2	Insert a capability. Returns {:error, {:duplicate_name, name}} if a capability with that name already exists.
unregister/2	Remove by name.
unregister_kind/2,3	Remove every capability of a kind, optionally filtered by a metadata match map. Used to clear capability families atomically: plugin tools on reload, sandbox preset commands on config change.
find/2	ETS lookup by name. Hot path; bypasses the GenServer.
list/1,2 / list_for/1,2	List all capabilities, with optional filtering.
refresh/2	Currently a no-op for every kind. Real re-syncs go through the owning components: plugin reload, skill_reload, and MCP connect/disconnect.
resolve_policy/2	Translate (trust, policy) into the effective allow/deny list. Public API.

Built-in tools

Thirty-five built-in tools ship unconditionally with Fermix, plus three bridges for deferred tools (tool_search, tool_describe, tool_call) when tool-schema deferral is enabled (on by default). Deferral, explained below, keeps rarely-used tool definitions out of each request to save tokens. Each implements the Builtin.Tool behaviour. One more built-in, computer_use, is seeded conditionally: it is experimental and off by default, and registers only when computer use is enabled in config and its native helper is installed (see the table row and the note below).

Name	Policy class	Category	What it does
shell	exec	system	Run a shell command via CommandRunner with sandbox-resolved working directory and env.
file_read	read_only	file	Read a UTF-8 text file, gated by the sandbox read-path check.
file_write	read_write	file	Write a file, gated by the sandbox write-path check.
file_edit	read_write	file	Replace an exact-string span in a file.
glob_search	read_only	file	Pattern-glob the workspace.
content_search	read_only	file	Regex-search file contents.
git_read	read_only	git	Run read-only git queries: status, log, diff, branch, show.
git_write	read_write	git	Run mutating git commands: add, commit, checkout, pull. Pushing is deliberately excluded.
web_fetch	network	web	Fetch a public URL through Net.Guard, with a body cap and IP pinning. Returns the page’s readable text, or the JSON verbatim when the endpoint serves JSON.
web_search	network	web	Web search with DuckDuckGo as the default keyless backend; six configurable backends (Brave, Exa, Tavily, Parallel, Perplexity, Firecrawl) are supported when set up. A hard error on a configured backend degrades once to DuckDuckGo, loudly.
browser	network	web	Drive a supervised local browser for JavaScript-rendered pages, forms, and interactive content. The screenshot action returns the captured page as an image the model can see, not just a saved path; the managed browser caps live tabs and auto-closes the oldest non-active tab past the cap.
memory_recall	read_only	memory	Look up a stored memory by key, list this conversation’s memories, or search stored memories and past conversation history.
memory_store	read_write	memory	Persist a memory candidate through the admission pipeline.
memory_sources_list	read_only	memory	List configured memory sources.
subagents	external_api	delegation	Run one or more temporary subagents for delegated work, concurrently up to a cap.
skill_create	read_write	skill_admin	Scaffold a new local Fermix skill with SKILL.md and starter evals.
skill_reload	read_write	skill_admin	Reload a skill from disk without restarting the daemon.
skill_view	exec	skill_admin	Show an installed skill’s instructions and metadata.
skill_run	exec	skill_admin	Run an installed skill by name.
skill_list	read_only	system	List installed skills available to the agent.
schedule_job	read_write	scheduling	Create a cron-style scheduled agent run. The effective capability policy comes from the creator’s trust level, not the model’s.
update_job	read_write	scheduling	Update a scheduled job’s schedule, prompt, or config.
list_jobs	read_only	scheduling	List existing scheduled jobs.
list_job_runs	read_only	scheduling	List past runs for a scheduled job.
get_job_run	read_only	scheduling	Fetch detail for one scheduled job run.
run_job_now	read_write	scheduling	Trigger an immediate out-of-schedule run of a job.
pause_job	read_write	scheduling	Pause a scheduled job.
resume_job	read_write	scheduling	Resume a paused scheduled job.
remove_job	read_write	scheduling	Delete a scheduled job.
model_routing_config	read_write	config	Read or update model routing config (provider, model, effort).
generate_image	external_api	media	Create or edit a raster image from a text prompt. The result is written under the sandbox media/ floor and sent to the current chat automatically. Backend is openai, openai_codex, xai, or google, configured via [fermix_core.tools.generate_image]. openai_codex runs on a connected ChatGPT/Codex login instead of an image API key — billed to that subscription. It is opt-in and fails loudly if the login is missing or not entitled; it never silently falls back to another backend.
tool_help	read_only	system	Return full documentation for one registered capability.
tool_search	read_only	system	Keyword search (using BM25, a standard text-relevance ranking) over deferred plugin/MCP tool schemas by description. Registered when tool-schema deferral is enabled (default on).
tool_describe	read_only	system	Fetch the full schema for one deferred tool by name. Registered when tool-schema deferral is enabled (default on).
tool_call	read_only	system	Invoke a deferred tool by name; the loop unwraps it before policy and dispatch, so the real tool’s own policy class is what gets enforced. Registered when tool-schema deferral is enabled (default on).
request_directory_access	external_api	system	Ask the owner, in chat, to approve access to a directory the sandbox currently denies. Surfaces only on attended operator turns; the owner confirms with a single-use 60-second /confirm token, the grant persists to [sandbox] allowed_roots, and on chat channels the original request resumes automatically. Guest, scheduled, and delegated runs never see it, and unsafe roots (all of $HOME, ~/.ssh, OS roots) are refused before the owner is even prompted. See sandbox.
send_attachment	read_only	channel	Send a local file through the active channel reply port. URLs are not fetched.
react	read_only	channel	React to the user’s message with a single fitting emoji instead of a text reply, for pure acknowledgements (“ok”, “thanks”, 👍). Advertised only on turns whose channel supports message reactions (on restricted channels the emoji choice is constrained to the channel’s allowed set); a delivered reaction with no accompanying text ends the turn without a further model call.
computer_use	gui_control	computer	Drive the host desktop by screenshot and mouse/keyboard, one action per call. Experimental and off by default. Registers for the model only when computer use is enabled and the native helper is installed; operator-only and never delegated to subagents.

The browser tool controls a supervised local browser process; web automation belongs there. The computer_use tool is a separate, experimental capability that drives the host desktop itself — the real mouse and keyboard of the logged-in session, guided by screenshots. It is off by default, and it registers for the model only when two things hold: computer use is enabled in config and the native helper is installed. The setup Plugins page (the “Computer Use” card) does both — it sets [fermix_core.computer_use] enabled = true and fetches the helper. The helper is fetched from the compux project’s own signed release and verified against a SHA-256 checksum baked into your Fermix build, then cached under ~/.fermix/plugins/compux/; fermix plugins does not manage it. You can also set the config key yourself: a daemon that has computer use enabled but is missing the matching helper downloads it on its next start (daemon boot only, bounded to about 30 seconds, and fail-soft — a failed fetch leaves computer use off until the next restart or the setup card). Supported on Apple Silicon (M-series) macOS and Linux x86_64. Config changes apply on the next daemon restart. Operating-system permissions (macOS Screen Recording and Accessibility) are a separate diagnostic, not part of that gate — fermix doctor’s computer-use check reports what is missing, so an ungranted permission informs the operator rather than silently hiding the tool. How far it may act follows the sandbox mode (strict is look-only), and a session only starts from an attended origin — an interactive chat, fermix ask, or voice — never from a scheduled job. Before any action that would move the pointer or type — clicking, dragging, scrolling, moving the mouse, typing, pressing a key, or pasting — Fermix checks how long the machine has been idle. If you are actively using it, the agent waits a few seconds for a gap, then either takes its turn or holds the action back and tells you it stepped aside. Idle detection is macOS-only. Where it is unavailable the action proceeds, so this is a courtesy that keeps the agent from fighting you for the cursor, not a safety boundary. To take the machine back yourself, /pause hands the cursor and keyboard back to you: the computer-use session stays alive and refuses further actions until you /resume — unlike /stop, which tears the session down. See computer use for the full platform, permission, action-loop, and safety model.

For per-tool parameter schemas, failure modes, and usage examples, see the tool reference.

Tool categories

Categories let a runtime context exclude a class of tools without naming each one individually (excluded_categories: [:scheduling]).

Category	Tools
:system	shell, tool_help, skill_list, tool_search, tool_describe, tool_call, request_directory_access, sandbox preset commands
:file	file_read, file_write, file_edit, glob_search, content_search
:git	git_read, git_write
:web	web_fetch, web_search, browser
:media	generate_image
:memory	memory_recall, memory_store, memory_sources_list
:scheduling	schedule_job, update_job, list_jobs, list_job_runs, get_job_run, run_job_now, pause_job, resume_job, remove_job
:delegation	subagents
:skill_admin	skill_create, skill_reload, skill_view, skill_run
:config	model_routing_config
:channel	send_attachment, react
:computer	computer_use
:plugin	Every tool registered by a ready plugin

How tools are presented to the model

MainAgent fetches the filtered capability list for the current trust level and conversation context, then formats each capability’s JSON Schema into the provider-specific tool-call format before passing it to AgentLoop. The loop calls the provider, parses any tool calls in the response, looks up each capability in the registry by name, and dispatches to the executor. Results are appended to the message list and the loop continues until the model issues a final text response or the iteration cap is reached (default 100 for interactive, sub-agent, and scheduled-job turns; these caps are internal system settings, not config.toml keys).

Sandbox enforcement applies to every shell call regardless of how it reaches the executor. See sandbox for path grants, env allow/deny, and command presets.

Tool-schema deferral

By default, the full definitions (schemas) for plugin and MCP tools are left out of the request sent to the provider to save tokens; only the tool names remain listed in the runtime prompt. The model fetches a full definition on demand when it actually needs that tool. Three bridge tools handle these deferred tools:

Bridge	What it does
tool_search	BM25 search over the deferred catalog to find tools by description.
tool_describe	Fetch the full schema for one deferred tool by name.
tool_call	Invoke a deferred tool by name; the loop unwraps it so traces and policy see the real tool name. Direct calls by name also work.

Deferral is on by default. To disable it, set [fermix_core.tools.tool_search] enabled = false in config.toml. With deferral off, all schemas are included inline on every call.

Skills and MCP as capability sources

Skills are not registered as provider-visible tools. They live in their own SkillRegistry; the model discovers them through a compact skill catalog in the runtime prompt and interacts with them via the skill_view and skill_run built-ins. Each skill’s SKILL.md still defines its own allowed-tool boundary (allowed_tools), so a skill’s sub-turns can be restricted to a subset of built-ins. Bundled and locally installed skills run at operator trust; plugin-shipped skills run at guest trust. A skill’s name may not collide with a registered capability name.

Ready plugins register their tools directly into the same registry (as :builtin kind, tagged plugin_owned? in metadata so they can be swapped atomically on plugin reload). A plugin that runs a local MCP process contributes its tools through MCP discovery instead, named <plugin>_<tool>.

Outbound MCP tools enter as :external_api policy class by default (overridable per tool with policy_class in [mcp.servers.<server>.tools.<tool>]) and are removed when the MCP connection drops. Operators can set hidden_from_agent = true on any MCP tool in config.toml to keep it configured but invisible to the model.

See skills, plugins, and MCP for setup and configuration details.