Operate

Everyday Operations

Name: InnerWarden
Author: InnerWarden

Day-to-day operator tasks for InnerWarden: check status, tame noisy detection, and manage the sensor, agent, and supervisor services.

Everyday Operations

For: operators running InnerWarden day to day, checking status, taming noise, and keeping the two services healthy.

Once InnerWarden is installed, most days you do not touch it. It watches, it explains, and (once you arm it) it acts. This page is the short list of things you actually do as an operator: see what it is doing, quiet down anything noisy, and understand the two background services that keep it alive.

If you are dealing with a live alert right now, go to Responding to Incidents. If you are still installing, start at Install and First Run.

The daily check

One command tells you whether the stack is healthy and what it saw today:

innerwarden get status

You will see the sensor and agent state, which capabilities are active, and today's incident/decision counts. If anything looks off, the next command tells you why and how to fix it:

innerwarden system doctor

doctor runs health checks and prints concrete fix hints, not just red marks. Read its output back in plain terms and act on the hints before changing anything by hand. It exits non-zero when there is a real problem, so it also works in a cron or a CI smoke check.

To see recent activity:

innerwarden get incidents --days 7        # what was detected
innerwarden get decisions --days 7        # what the agent decided to do about it
innerwarden get sensors                   # which collectors and eBPF programs are live
innerwarden get posture                   # baseline-learning status, what "normal" looks like here

Every flag for these lives in CLI Reference; this page just points you at the useful ones.

Watching live

When you want a real-time tail instead of a snapshot:

innerwarden stream --type incidents       # live incident feed
innerwarden stream --type events           # live raw event feed (chatty)

Most operators prefer the dashboard for live work because it groups and explains as it goes. See Dashboard and Notifications.

When it is too noisy

A fresh install is deliberately curious: it flags first and learns what is normal on your specific host over a learning window. If detection is too chatty, do not reach for blanket suppression. Two safe knobs, in order:

innerwarden system calibrate              # tune detector thresholds to THIS host's normal
innerwarden config sensitivity <level>    # dial overall detection/notification chattiness up or down
innerwarden system tune --days 7          # propose threshold changes from recent history (asks before applying)

calibrate and tune look at what your box actually does and adjust thresholds to it. config sensitivity is the coarse master dial. None of these blind-trusts anything.

For a specific benign source that keeps firing (a known scanner you run, a backup IP, a service account), do not lower sensitivity for everyone. Verify it is genuinely benign first, then trust just that one entity. That whole verify-then-trust workflow has its own page because getting it wrong hands an attacker a free pass: Safe Observe and Allowlist.

Trusting an entity means monitor-only: it is still detected, logged, and notified, only the automatic block is suppressed. You keep visibility even on things you trust. That is deliberate.

The two services

InnerWarden is two Rust processes, by design:

innerwarden-sensor does deterministic collection. It runs the eBPF programs, reads logs, captures network metadata, and runs the detectors. It makes no AI calls and opens no outbound HTTP. If it sees something, it writes an event or an incident. That is all it does, on purpose, so the part with kernel-level access stays simple and auditable.
innerwarden-agent is the interpretive layer. It reads the sensor's incidents, triages them, correlates across layers, decides on a response, runs the dashboard, and sends notifications. It is the part allowed to call out (to an AI provider, an IP-reputation feed, Telegram).

What each part does in depth lives in Architecture.

Managing the services

# Linux (systemd), a licensed install: the watchdog runs the agent
sudo systemctl restart innerwarden-sensor
sudo systemctl restart innerwarden-watchdog     # restarts the agent it runs
sudo journalctl -u innerwarden-watchdog -f      # the agent's log is in this unit

# Linux (systemd), an agent running as its own unit
sudo systemctl restart innerwarden-agent
sudo journalctl -u innerwarden-agent -f

The host stack (sensor and agent) runs on Linux only. On macOS and Windows you run the free Community guardrail, which installs no background service to manage.

Stopping or restarting the watchdog stops the agent once: the agent gets SIGTERM, then SIGKILL if it is still running 4 seconds later, and nothing is respawned after a stop. An agent that exits while nobody asked for a stop is respawned, however it exited, so signalling the agent cannot end protection. Repeated crashes wait longer each time (200 ms, doubling up to 30 seconds), and the wait resets after a minute of uptime.

One caution: when a supervisor or the watchdog is running (a licensed install runs the watchdog by default), it spawns the agent as its own child. On those boxes do not systemctl start innerwarden-agent by hand, you would end up with two agents fighting over the dashboard port and the data directory. See the supervisor section below.

The supervisor (crash recovery)

innerwarden-supervisor is the standard resilience layer that keeps the agent alive. It replaces a plain Restart=on-failure with something that actually understands the agent:

a periodic HTTP health probe against the agent's /livez endpoint,
force-kill plus respawn when the agent is alive but unresponsive (not just when it has crashed),
a per-hour restart rate limit so a crash loop cannot hammer the box,
a Telegram alert on every restart, when a token and chat id are in its environment.

Because the supervisor spawns the agent as its own child, it must not run alongside the standalone innerwarden-agent.service, two agents would collide on the dashboard port and the data dir. Enabling the supervisor means disabling the agent unit:

sudo systemctl disable --now innerwarden-agent
sudo systemctl enable  --now innerwarden-supervisor

An example unit ships at examples/systemd/innerwarden-supervisor.service. It runs as the unprivileged innerwarden user (same uid and sudoers grants as the standalone agent), sources /etc/innerwarden/agent.env for API keys and tokens, and passes arguments through to the agent with --agent-arg. The probe URL is HTTPS by default (https://127.0.0.1:8787), because the agent serves TLS and loopback cert verification is auto-skipped, a plain http:// there would fail every probe and SIGKILL a healthy agent in a loop, so leave it on HTTPS.

The supervisor's own flags (--agent-binary, --agent-arg, --agent-api, --health-interval, --max-restarts-per-hour, --log-level) are documented in CLI Reference.

Supervisor vs. the proprietary watchdog

Same engine, two products:

The supervisor gives every install resilience: auto-restart on its own.
The separate proprietary innerwarden-watchdog wraps that same supervisor and adds the anti-tamper layer, process stealth, SHA-256 integrity gating (it refuses to respawn a swapped-out binary), and a license check.

Run one or the other. Never run both the supervisor and the standalone agent unit at the same time.

Permissions

The installer creates a dedicated, unprivileged innerwarden service user and grants it only what it needs. You normally never touch this, but here is the shape of it for a manual or hardened setup on Linux:

sudo useradd -r -s /sbin/nologin innerwarden        # dedicated service user
sudo usermod -aG systemd-journal innerwarden        # read the journal without root
sudo usermod -aG docker innerwarden                 # read the Docker socket (container monitoring)
sudo mkdir -p /var/lib/innerwarden
sudo chown innerwarden:innerwarden /var/lib/innerwarden

The person who ran the installer is added to the innerwarden group, only so the free guardrail can write its records into the agent's data directory. That group never reads a secret, and the operator is often the user an AI agent runs as, so the secrets in /etc/innerwarden/ are closed to it:

File	Owner and mode	Who reads it
`agent.env` (AI key, chat tokens, webhook URLs, dashboard password hash)	`root:root 0600`	root: systemd hands it to the services, the sensor hashes it, `innerwarden-ctl` reads it under `sudo`
`license.key`	`root:innerwarden-secrets 0640`	the agent, the watchdog, `innerwarden-config-sign`, and the DNS guard through a unit drop-in; the installer puts only the `innerwarden` service user in `innerwarden-secrets`, never the operator
`agent.toml`	owned by the `innerwarden` service user, `0600`	the agent, as the service user

Where the innerwarden-secrets group cannot be made, the licence stays root-only (0600). Re-running the installer, or sudo innerwarden-ctl upgrade (even when already up to date), puts an older install's files back to these modes.

Response skills that touch the firewall need a narrow, command-scoped sudoers grant for the backend you use. The installer writes the right one; the grant is deliberately limited to the exact command (for example ufw deny from *, or the matching iptables / nft / firewall-cmd rule), never a blanket sudo. The precise drop-in per backend is shown in Trust and Safety Invariants.

Rule that catches people: data_dir in your config must match ReadWritePaths in the systemd unit. If they disagree, the agent cannot write its state and the service flaps.

The kernel-level capabilities (eBPF, raw network capture) are why install needs sudo; the running services keep the minimal set, not full root, except where the watchdog deliberately runs the agent as a root child on hardened production boxes.

Upgrades

Upgrades are toolchain-free and signature-verified, no compiler on the box:

innerwarden-ctl upgrade --check           # see if a newer release exists
sudo innerwarden-ctl upgrade               # fetch + verify (SHA-256 / signature) + install the latest

The upgrade keeps your config and data. Re-running the installer over an existing install is also an upgrade: it stops a running watchdog (and the agent it runs) before it replaces any binary, and starts it again on the new binaries, so the watchdog never finds its agent swapped underneath it. The sensor keeps running throughout. To check the host stack release you are on at any time:

innerwarden-ctl --version

(innerwarden --version is the free Community CLI's release, which moves on its own schedule.)

A renewed licence is fetched and installed with sudo innerwarden-config-sign license refresh; the installer also enables innerwarden-license-refresh.timer, which runs it daily. When the licence is 14 days or fewer from expiry, the licence check at startup logs a notice naming that command, and the error for an expired licence names it too. A refresh keeps the licence's owner, group and mode, so the DNS guard keeps reading it.

Removing InnerWarden is a single command (sudo innerwarden-ctl uninstall, --purge to also drop config and data; innerwarden uninstall would remove the free CLI instead); the command and its flags are owned by CLI Reference, and Install and First Run links it.

Where things live

Path	What it is
`/etc/innerwarden/`	config (`agent.toml`, `sensor` config, `agent.env` secrets)
`/var/lib/innerwarden/`	data: incidents, decisions, telemetry, baseline, audit trail, models
`/var/log/innerwarden/`	logs (on setups that route there)
`examples/systemd/`	the shipped unit files, including the supervisor unit

All config keys, including data retention, are documented in Configuration; the retention schedule itself and how to export or erase data are owned by Privacy and GDPR.