Operate

Configuration Reference

Name: InnerWarden
Author: InnerWarden

The full TOML and environment-variable reference for the InnerWarden sensor and agent, organised by area, with safe defaults explained.

For: operators - the canonical TOML and environment-variable reference, organised by area.

InnerWarden has two config files: one for the sensor (/etc/innerwarden/config.toml, what to collect and detect) and one for the agent (/etc/innerwarden/agent.toml, how to triage, notify, and respond). Both have sane defaults, so a fresh install runs fine with almost nothing set. You only edit these when you want to change behaviour.

You almost never need to hand-edit. The innerwarden config <...>, innerwarden trust, and innerwarden-ctl setup commands write these keys for you and re-check them. This page is the reference for what each key means; the commands that set them live on CLI Reference. After any hand-edit, run innerwarden config validate (it fails loudly on a typo'd or unknown key, before the agent boots).

A note that saves confusion: the responder ships off and dry-run on a fresh install. Arming enforcement is a deliberate step. Read Safe Observe and Allowlist before you set responder.dry_run = false.

Minimum useful config

Most installs need only a handful of keys. Everything else has a working default. A typical small server:

# agent.toml
[ai]
enabled = true
provider = "openai"          # or "ollama" for local, or set [ai.warden] for on-device
model = "gpt-4o-mini"

[telegram]
enabled = true               # operator alerts on your phone

[responder]
enabled = false              # start OFF; arm later, see Safe Observe and Allowlist
dry_run = true               # SAFETY: always starts true
block_backend = "nftables"   # written by the installer for this host (see Responder and safety)

# config.toml (sensor)
[agent]
host_id = "my-server"        # the name that appears on every alert

[output]
data_dir = "/var/lib/innerwarden"

That is enough. The rest of this page is the full menu, by area.

Sensor (`config.toml`)

The sensor decides what to collect and which detectors run. It is deterministic: no AI, no HTTP. Most detectors are on sensible defaults; the advanced ones default to off so you can enable them after the host has a baseline.

Core

[agent]
host_id = "my-server"        # appears on every alert and report

[output]
data_dir = "/var/lib/innerwarden"
write_events = true
# redis_url = "redis://127.0.0.1:6379"   # optional: stream events/incidents to Redis Streams
# redis_stream = "innerwarden:events"
# redis_maxlen = 50000                    # approximate cap via MAXLEN ~

Collectors

Each collector reads one source. The example is what the installer writes on Linux.

[collectors.auth_log]
enabled = true
path = "/var/log/auth.log"

[collectors.journald]
enabled = true
units = ["sshd", "sshd-session", "sudo"]   # "sshd" not "ssh"; "sshd-session" is where newer OpenSSH logs logins

[collectors.docker]
enabled = true

[collectors.integrity]
enabled = true
poll_seconds = 60
paths = ["/etc/ssh/sshd_config", "/etc/sudoers"]

[collectors.exec_audit]
enabled = false
path = "/var/log/audit/audit.log"
include_tty = false          # high privacy impact; enable only with explicit authorisation

[collectors.syslog_firewall]
enabled = false              # alternative to journald.kernel for hosts without journald
path = "/var/log/syslog"     # feeds the port_scan detector

Debian 12 and later, RHEL (which logs to /var/log/secure instead) and most minimal images have no /var/log/auth.log. There the journald collector carries SSH and sudo, and the sensor runs on without the file.

The native network capture (DNS over UDP:53, HTTP over TCP:80, and TLS JA3/JA4) plus the eBPF programs need no config beyond the kernel capabilities the installer grants. There are no external log-reader collectors (no Falco/Suricata/Wazuh/osquery ingest); the native eBPF + raw-packet coverage is the surface. If you must ingest another tool's logs, see Integration Recipes.

Detectors

A few detectors are on by default; the rest default to enabled = false so you can switch them on once the host's normal traffic is known. Each typically takes a threshold and a window. What each one catches is on What It Detects.

# On by default
[detectors.ssh_bruteforce]
enabled = true
threshold = 8
window_seconds = 300

[detectors.setns_owner]
enabled = true
cooldown_seconds = 300       # root joining a non-root-owned user namespace

[detectors.untrusted_root_exec]
enabled = true
cooldown_seconds = 300       # uid-0 exec from an unprivileged-writable path

[detectors.reverse_shell]
enabled = true               # Critical, judged by the shell's shape, never by who started it

[detectors.ransomware]
enabled = true

[detectors.dns_tunneling]
enabled = true

[detectors.crypto_miner]
enabled = true

# Enable after a baseline (representative subset; see What It Detects for the full list)
[detectors.credential_stuffing]
enabled = false
threshold = 6                # distinct usernames per IP in window
window_seconds = 300

[detectors.port_scan]
enabled = false
threshold = 12               # unique destination ports per IP in window
window_seconds = 60

A set of detectors is always on and needs no config: docker_anomaly, integrity_alert, c2_callback, container_escape, distributed_ssh, suspicious_login, process_tree, and privesc, among others.

There is no key for the DNS exemption: c2_callback never counts port-53 traffic to the resolvers the host is configured with (/etc/resolv.conf, systemd-resolved) or to a documented cloud platform resolver such as Azure's 168.63.129.16; tunnelling through those resolvers is still caught. See What It Detects.

Prefer innerwarden system tune over editing thresholds by hand: it proposes adjustments from this host's recent noise and signal. To silence a single noisy-but-benign detector, use innerwarden trust suppress <pattern> rather than disabling it. Both are on CLI Reference.

Agent (`agent.toml`)

The agent is the interpretive layer: triage, correlation, response, and notifications. Every field has a default, so the file is optional.

AI provider (`config ai`)

Set by innerwarden config ai.

[ai]
enabled = true
provider = "openai"          # openai | anthropic | ollama | groq | deepseek | together |
                             # minimax | mistral | xai | gemini | fireworks | openrouter | azure_openai
# api_key = ""               # or env OPENAI_API_KEY / ANTHROPIC_API_KEY
model = "gpt-4o-mini"
# base_url = ""              # endpoint override (ollama + OpenAI-compatible providers); or env OLLAMA_BASE_URL
context_events = 20          # recent events sent to the AI as context
confidence_threshold = 0.85  # below this, no auto-execution
incident_poll_secs = 2
max_ai_calls_per_tick = 5    # 0 = unlimited
circuit_breaker_threshold = 0    # 0 = disabled; ~20 recommended for DDoS defense
circuit_breaker_cooldown_secs = 60

For on-device triage with no API key, install the Local Warden model (innerwarden install-warden) and it writes its own [ai.warden] section. AI is optional: the deterministic detectors plus the on-device model already give a working system.

Which model does what

The agent has two model roles, and an optional third for one screen:

Role	Where it is set	What it does
Local Warden (on-device)	`[ai.warden]`, written by `innerwarden install-warden`	Decides incidents. A narrow classifier trained for incident triage: it does not answer free-form prompts, so it cannot verify an observation.
Language model	`[ai]` (what `config ai` writes), or `[ai.llm]` to give this role its own provider	Verifies the observation queue, writes the case narrative ("Explain with AI"), and answers as the shell of an `llm_shell` honeypot.
Case narrative only	`[ai.explain]` (optional)	A stronger model for the case narrative alone.

When [ai.llm] is absent, [ai] fills the language-model role, so sudo innerwarden-ctl config ai <provider> is all most hosts need. You write [ai.llm] by hand only to give that role a different provider from [ai]. If an item waits for review with a reason that names [ai.llm], the host has no language model to verify with: run sudo innerwarden-ctl config ai <provider>.

A cloud provider only verifies, next to the Local Warden. On a host that has the Local Warden (the paid installer provisions it), the Local Warden decides incidents. A cloud provider you add does not take that over: it verifies the observation queue and writes narratives. Its verdict that an observation is suspicious labels the item for a person; it becomes a decision the model can act on only if you set [incident_flow] escalate_to_decide = true. That is off by default, deliberately: turn it on after you have read what the model would decide. A host with a cloud provider and no Local Warden has no other decider, so there the cloud model decides incidents too.

What the verifier asks, and what it accepts

The verifier sends the ambiguous items ([observation] ai_verification) in batches of ai_batch_size, as a numbered list, and tells the model to answer with a JSON array and nothing else. Each element has three fields:

[
  {"item": 1, "verdict": "NORMAL", "reason": "package manager post-install script"},
  {"item": 2, "verdict": "SUSPICIOUS", "reason": "shell started by a web server process"}
]

item: the item's number in the list, an integer starting at 1.
verdict: NORMAL or SUSPICIOUS, in any letter case.
reason: one line, recorded with the verdict.

The reader is forgiving about wrapping, because it takes everything from the first [ to the last ]: a code fence or a sentence around the array is fine. It is strict about the shape. A reply with no readable array, an item left out of it, an item number outside the batch, or a verdict that is neither word gives those items no verdict, and they go to a person: they wait in Cases as needs review, with the reason "AI verifier returned no verdict". A model or an OpenAI-compatible server that does not follow this format therefore shows up as every item waiting for review, not as an error. A NORMAL verdict never closes a critical finding or one in the threat floor either: it waits for a person, with the reason "a critical or threat-floor finding is never closed without a person".

Responder and safety (`config responder`)

The single most important block. Set by sudo innerwarden-ctl config responder --mode observe|dry-run|live.

[responder]
enabled = false              # default; a fresh install does not enforce
dry_run = true               # SAFETY: always starts true (log what it would do, do nothing)
block_backend = "nftables"   # nftables | ufw | iptables | firewalld | xdp; the installer picks one for the host
                             # firewalld = the RHEL/Rocky/CentOS/Fedora/openSUSE firewall
allowed_skills = ["block-ip-nftables", "monitor-ip"]

The installer chooses block_backend for the host it finds, and names the matching block-ip-<backend> skill in allowed_skills. From 0.16.75 it takes nftables (InnerWarden's own table, which a firewall reload leaves alone) whenever the nft command is installed. Without nft it takes a running ufw, then a running firewalld, then iptables, then a ufw or firewalld that is installed but not running. A host with no firewall at all is written as ufw, and sudo innerwarden-ctl arm then says there is no firewall to block through. Earlier releases preferred a running ufw over nftables. An upgrade keeps the backend that agent.toml already names.

dry_run = false is live enforcement. Do not set it before a clean baseline and a verified trust list, see Safe Observe and Allowlist and Trust and Safety Invariants.

Second factor (`config 2fa`)

Off on a fresh install. Turning it on asks for a 6-digit code before a change that outlives the incident, and leaves one-shot approvals as one tap. Set the method with innerwarden config 2fa; the per-action table is yours to change.

[security]
two_factor_method = "totp"   # "none" (default) or "totp"

[security.require_second_factor]
# Anything left out keeps its shipped default. Values: none | totp.
confirmation_always    = "totp"   # a standing auto-approval
gate_always_allow_path = "totp"   # a kernel allowlist entry, forever
mode_change            = "totp"   # whether the responder enforces for real
unblock                = "totp"   # reopening an address to the host
allowlist_add          = "totp"
allowlist_undo         = "totp"
detector_disable       = "totp"   # switching a control off
bulk_verdict           = "totp"   # "answer all like this" over many cases at once

# One tap by default, listed here so you can raise one if you want to:
# confirmation_approve, confirmation_reject, gate_allow_binary, gate_reject,
# gate_undo, quick_block, review_block, review_ignore, honeypot, detector_enable

gate_allow_binary is permanent by hash and still ships as one tap: it is what you reach for while a deploy is blocked. Raise it if your host is quieter than that.

A mistyped key or level fails the config rather than being ignored, because a setting that silently does nothing reads as protection that is not there. That includes link: a one-time confirmation link is not something this release can produce, so the agent refuses the word instead of quietly asking for a code and calling it a link.

Allowlist and trust (`trust add`)

Set by innerwarden trust add / trust remove. Trusted entities are still detected and notified; only their auto-block is suppressed.

[allowlist]
trusted_ips = []             # IPs or CIDRs that skip the AI gate, e.g. ["10.0.0.0/8", "192.168.1.100"]
trusted_users = []           # usernames that skip the AI gate

Never populate this blindly. The never-blind observe, verify, propose, arm workflow is owned by Safe Observe and Allowlist.

The agent guardrail's decisions (`guard_ingest`)

The seam between the two halves. The free guardrail appends one line per blocked command or MCP tool call to guard-events.jsonl; the host agent tails that file and turns each into an incident, so what an agent was stopped from doing reaches the graph, the correlation engine, and the model. Without it the host half is blind to everything the agent half refused.

[guard_ingest]
enabled = true
events_path = "/home/YOUR-USER/.config/innerwarden/guard-events.jsonl"

events_path must point at a person's home, not the agent's. The agent runs as the unprivileged innerwarden service user, so the built-in default, derived from that process's $HOME, resolves somewhere the free guardrail never writes. The installer resolves the real path and writes this section for you; set it by hand only when the guardrail runs under a different account than the one that installed.

Check it is working:

wc -l ~/.config/innerwarden/guard-events.jsonl   # the guardrail is writing
innerwarden get incidents --days 1               # and the host is reading

A file that is not growing while an agent is being refused means the two halves are not joined. The usual cause is a path pointing at the wrong home.

Correlation, observation, and learning

These tune how the agent stitches events into chains and how it handles ambiguous alerts. Sensible defaults; an absent section uses them.

[correlation]
enabled = true
window_seconds = 300
max_related_incidents = 8

[observation]
enabled = true
auto_dismiss_threshold = 70  # min score to auto-dismiss without AI
auto_escalate_threshold = 40 # max score to auto-escalate without AI
ai_verification = true       # send the ambiguous middle band to AI
ai_batch_size = 10
maintenance_windows = []     # ["HH:MM-HH:MM", ...] local time; items inside get a context bump

[learning]
suppression_mode = "off"     # off | shadow | enforce (shadow logs but changes nothing); a new install is written with off
min_dismissals = 5           # repeated operator dismissals before a pattern is eligible to auto-suppress
llm_escalation_enabled = true
llm_escalation_min_confidence = 0.75  # below this, a high-impact escalation waits for a human
needs_review_notify = false  # a Telegram message for every item left for you to review (off: they wait in Cases)
emit_labels = true

From 0.16.75 the installer writes the shadow experiments off on a new install: suppression_mode above, and decide_modifier_mode, fp_suppression_mode and graph_incidents_mode under [kg]. In shadow each one writes a dated log of what it would have done and changes nothing, which costs disk on a host where nobody reads it. To see what they would do, run sudo innerwarden-ctl system kg-shadow-report, then set a mode to shadow (measure) or enforce (act). An existing agent.toml keeps whatever it sets, and a file that sets none runs the code default, shadow.

Ambiguous alerts that score in the middle band wait for your decision rather than auto-acting. Low and medium parked items auto-resolve to "dismiss" after 24 hours; high and critical never auto-close and stay visible until a human acts. How you act on them is on Responding to Incidents.

Notifications

All set by the innerwarden config <channel> commands on CLI Reference; what they look like is on Dashboard and Notifications.

[telegram]
enabled = false
# bot_token = ""             # or env TELEGRAM_BOT_TOKEN
# chat_id = ""               # or env TELEGRAM_CHAT_ID
# daily_summary_hour = 8     # optional daily summary at this local hour (0-23)

[slack]
enabled = false
# webhook_url = ""           # or env SLACK_WEBHOOK_URL
min_severity = "high"
# dashboard_url = ""         # optional deep-link in messages

[discord]
enabled = false
# webhook_url = ""           # or env DISCORD_WEBHOOK_URL
min_severity = "high"

[webhook]
enabled = false
url = "https://hooks.example.com/notify"
min_severity = "medium"      # debug | info | low | medium | high | critical
                             # every incident at or above it is posted, as it happens,
                             # whatever the chat channels' notification gate decides
timeout_secs = 10

[web_push]
enabled = false
min_severity = "high"
# VAPID keys are generated by: innerwarden config web-push

A chat channel's min_severity is not a list of what it posts. Telegram and Slack page you when an active intrusion is not contained, at or above that level; every other incident goes to the daily briefing. A brute force that was blocked, or a port scan, does not page, in monitor-only or armed: What pages you by default lists what does and does not. The webhook is the channel that posts every incident at or above its min_severity. sudo innerwarden-ctl config sensitivity sets the level for Telegram and the webhook together.

Dashboard

Bind and access are best managed with innerwarden-ctl dashboard (it writes [dashboard] bind and restarts the agent safely); see CLI Reference and Dashboard and Notifications. The remaining knobs:

[dashboard]
# trusted_proxies = ["127.0.0.1", "::1"]
session_timeout_minutes = 480  # 8h inactivity timeout for Bearer sessions
max_sessions = 5               # max concurrent sessions; oldest evicted when exceeded

Enrichment integrations

[abuseipdb]
enabled = false
# api_key = ""               # or env ABUSEIPDB_API_KEY
auto_block_threshold = 0     # 0 = disabled; 80+ recommended for known-botnet blocking

[dshield]
enabled = false              # SANS Internet Storm Center, keyless, read-only IP reputation

[geoip]
enabled = false              # ip-api.com, no key

[fail2ban]
enabled = false              # polls fail2ban-client CLI; Linux only

[cloudflare]
enabled = false
# zone_id = ""
# api_token = ""             # or env CLOUDFLARE_API_TOKEN
auto_push_blocks = true
block_notes_prefix = "InnerWarden:"

[crowdsec]
enabled = false              # EXPERIMENTAL: large community lists need ipset/nftables backend
# lapi_url = "http://localhost:8080"
# api_key = ""               # or env CROWDSEC_API_KEY
poll_secs = 60
max_per_sync = 50            # cap new IPs blocked per tick (avoids OOM on large lists)

Mesh (collaborative defense)

Set by innerwarden config mesh / innerwarden mesh connect.

[mesh]
enabled = false
bind = "0.0.0.0:8790"
poll_secs = 30
auto_broadcast = true
max_signals_per_hour = 50
initial_trust = 0.7          # trust level for configured peers (0.0-1.0)

# [[mesh.peers]]
# endpoint = "https://peer1:8790"
# public_key = ""            # may be empty; discovered via /mesh/ping at startup
# label = "dc-east"

Honeypot

A behavioural trap. Off-by-cost by default (banner mode just responds to probes). The interaction modes:

`interaction`	What it does
`banner` (default)	Sends a fake banner, captures connection metadata, never authenticates. Lowest cost, least signal.
`medium`	Captures every credential attempt and rejects all of them. No shell ever opens.
`llm_shell`	Tiered SSH auth, then an LLM-backed fake shell that records every command. Catches Mirai-class bots and human attackers. Opt-in. The model only writes the decoy's text (below).

In llm_shell the model only writes decoy text. Its job is the reply the attacker sees to each command. It is given no tools and cannot act: the attacker's commands are recorded and answered, never run on your host, and the reply is cleaned before it is sent so it does not carry real details of the machine. What the model writes goes to the attacker's terminal and to the session record, nowhere else.

[honeypot]
mode = "demo"                # demo | listener | always_on
bind_addr = "127.0.0.1"      # "0.0.0.0" + allow_public_listener for a production trap
port = 2222
http_port = 8080
services = ["ssh"]           # ["ssh", "http"] for multi-service
interaction = "banner"       # banner | medium | llm_shell
allow_public_listener = false
max_connections = 64
forensics_keep_days = 7
on_ai_decision = "ask"      # ask | automatic (see below)

When the AI decides "honeypot" for an attacker. With on_ai_decision = "ask" (the default) you get a four-button choice on Telegram, unless the model itself was confident enough to act. With "automatic" nobody is asked: the action runs, the decision is written to the audit trail, and you read it later in the cases and the reports. A misspelling fails the config rather than quietly meaning "ask". A prompt nobody answers within the approval window falls back to the model's stated alternative (block, or monitor) and says so in the audit trail; before 0.16.61 it was dropped and the attacker got nothing.

On an always_on host, "honeypot" means redirect. The listener is already running, so the action does not start a second one (which the isolation guard would refuse). It adds a per-IP redirect of port 22 into the listener port for duration_secs, then removes it. That path needs [honeypot.redirect] enabled = true; with it off, the decision is recorded as skipped and tells you which key to set.

The full honeypot block (sandbox, containment, external handoff, redirect) has many more keys; run innerwarden config validate after editing and see the in-file comments. The weak-credential list used by llm_shell mode is compiled in by design.

SOC playbooks

Operator-authored response runbooks. Disabled by default so a fresh install never auto-runs block/suspend steps. Built-ins are always embedded; list them with innerwarden rule list --type playbooks.

[playbooks]
enabled = false              # master switch
rules_dir = "/etc/innerwarden/rules/playbooks"
shadow = false               # when enabled, run for the audit trail only (no skill fires)

Host tags

[agent]
tags = []                    # e.g. ["env=prod", "role=web"]; scope playbooks to a host role

Data retention

Retention is owned by Privacy and GDPR (it documents both the schedule and the GDPR export/erase contract). The keys live in agent.toml:

[data]
events_keep_days = 7
incidents_keep_days = 30
decisions_keep_days = 90
telemetry_keep_days = 14
reports_keep_days = 30

Environment variables

Secrets and overrides. Put them in /etc/innerwarden/agent.env (or .env locally); it loads at startup and is fail-silent if absent. Anything set in TOML can also come from the matching env var.

agent.env is root:root 0600: systemd reads it as root and hands it to the services as their environment, so nothing else needs to open it, and the operator's innerwarden group cannot. Edit it with sudo. The innerwarden config writers create it owner-only and keep it that way, and re-running the installer or innerwarden-ctl upgrade closes a copy left readable by an older install.

# AI providers
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
OLLAMA_BASE_URL=http://localhost:11434   # optional override

# Notifications
TELEGRAM_BOT_TOKEN=<id>:<secret>
TELEGRAM_CHAT_ID=<numeric>
SLACK_WEBHOOK_URL=https://hooks.slack.com/services/...
DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/...

# Enrichment
ABUSEIPDB_API_KEY=...
CLOUDFLARE_API_TOKEN=...
VT_API_KEY=...                           # or VIRUSTOTAL_API_KEY; threat-feed hash checks
# DShield and GeoIP are keyless: enable via [dshield] / [geoip], no env var needed

# Dashboard auth
INNERWARDEN_DASHBOARD_USER=admin
INNERWARDEN_DASHBOARD_PASSWORD_HASH=$argon2id$...   # generated by innerwarden config dashboard

# Debug
RUST_LOG=innerwarden_agent=debug

Where to go next

The commands that write all of these keys: CLI Reference.
Arming enforcement safely, the never-blind allowlist: Safe Observe and Allowlist.
Systemd services, the supervisor, file permissions, and where data lives: Everyday Operations.
What each detector catches: What It Detects.
Retention and data-subject requests: Privacy and GDPR.