Operate

Everyday Operations

Name: InnerWarden
Author: InnerWarden

Day-to-day operator tasks for InnerWarden: check status, tame noisy detection, and manage the sensor, agent, and supervisor services.

Everyday Operations

For: operators running InnerWarden day to day, checking status, taming noise, and keeping the two services healthy.

Once InnerWarden is installed, most days you do not touch it. It watches, it explains, and (once you arm it) it acts. This page is the short list of things you actually do as an operator: see what it is doing, quiet down anything noisy, and understand the two background services that keep it alive.

If you are dealing with a live alert right now, go to Responding to Incidents. If you are still installing, start at Install and First Run.

The daily check

One command tells you whether the stack is healthy and what it saw today:

innerwarden get status

You will see the sensor and agent state, which capabilities are active, and today's incident/decision counts. If anything looks off, the next command tells you why and how to fix it:

innerwarden system doctor

doctor runs health checks and prints concrete fix hints, not just red marks. Read its output back in plain terms and act on the hints before changing anything by hand. It exits non-zero when there is a real problem, so it also works in a cron or a CI smoke check.

To see recent activity:

innerwarden get incidents --days 7        # what was detected
innerwarden get decisions --days 7        # what the agent decided to do about it
innerwarden get sensors                   # which collectors and eBPF programs are live
innerwarden get posture                   # baseline-learning status, what "normal" looks like here

Every flag for these lives in CLI Reference; this page just points you at the useful ones.

Watching live

When you want a real-time tail instead of a snapshot:

innerwarden stream --type incidents       # live incident feed
innerwarden stream --type events           # live raw event feed (chatty)

Most operators prefer the dashboard for live work because it groups and explains as it goes. See Dashboard and Notifications.

When it is too noisy

A fresh install is deliberately curious: it flags first and learns what is normal on your specific host over a learning window. If detection is too chatty, do not reach for blanket suppression. Two safe knobs, in order:

innerwarden system calibrate              # tune detector thresholds to THIS host's normal
innerwarden config sensitivity <level>    # dial overall detection/notification chattiness up or down
innerwarden system tune --days 7          # propose threshold changes from recent history (asks before applying)

calibrate and tune look at what your box actually does and adjust thresholds to it. config sensitivity is the coarse master dial. None of these blind-trusts anything.

For a specific benign source that keeps firing (a known scanner you run, a backup IP, a service account), do not lower sensitivity for everyone. Verify it is genuinely benign first, then trust just that one entity. That whole verify-then-trust workflow has its own page because getting it wrong hands an attacker a free pass: Safe Observe and Allowlist.

Trusting an entity means monitor-only: it is still detected, logged, and notified, only the automatic block is suppressed. You keep visibility even on things you trust. That is deliberate.

The two services

InnerWarden is two Rust processes, by design:

innerwarden-sensor does deterministic collection. It runs the eBPF programs, reads logs, captures network metadata, and runs the detectors. It makes no AI calls and opens no outbound HTTP. If it sees something, it writes an event or an incident. That is all it does, on purpose, so the part with kernel-level access stays simple and auditable.
innerwarden-agent is the interpretive layer. It reads the sensor's incidents, triages them, correlates across layers, decides on a response, runs the dashboard, and sends notifications. It is the part allowed to call out (to an AI provider, an IP-reputation feed, Telegram).

What each part does in depth lives in Architecture.

Managing the services

# Linux (systemd)
sudo systemctl restart innerwarden-sensor
sudo systemctl restart innerwarden-agent
sudo journalctl -u innerwarden-agent -f

# macOS (launchd)
sudo launchctl kickstart -k system/com.innerwarden.sensor
sudo launchctl kickstart -k system/com.innerwarden.agent

One caution on production hardening setups: when a supervisor (or the proprietary watchdog) is running, it spawns the agent as its own child. On those boxes do not systemctl start innerwarden-agent by hand, you would end up with two agents fighting over the dashboard port and the data directory. See the supervisor section below.

The supervisor (crash recovery)

innerwarden-supervisor is the standard resilience layer that keeps the agent alive. It replaces a plain Restart=on-failure with something that actually understands the agent:

a periodic HTTP health probe against the agent's /livez endpoint,
force-kill plus respawn when the agent is alive but unresponsive (not just when it has crashed),
a per-hour restart rate limit so a crash loop cannot hammer the box,
a Telegram alert on every restart, when a token and chat id are in its environment.

Because the supervisor spawns the agent as its own child, it must not run alongside the standalone innerwarden-agent.service, two agents would collide on the dashboard port and the data dir. Enabling the supervisor means disabling the agent unit:

sudo systemctl disable --now innerwarden-agent
sudo systemctl enable  --now innerwarden-supervisor

An example unit ships at examples/systemd/innerwarden-supervisor.service. It runs as the unprivileged innerwarden user (same uid and sudoers grants as the standalone agent), sources /etc/innerwarden/agent.env for API keys and tokens, and passes arguments through to the agent with --agent-arg. The probe URL is HTTPS by default (https://127.0.0.1:8787), because the agent serves TLS and loopback cert verification is auto-skipped, a plain http:// there would fail every probe and SIGKILL a healthy agent in a loop, so leave it on HTTPS.

The supervisor's own flags (--agent-binary, --agent-arg, --agent-api, --health-interval, --max-restarts-per-hour, --log-level) are documented in CLI Reference.

Supervisor vs. the proprietary watchdog

Same engine, two products:

The supervisor gives every install resilience: auto-restart on its own.
The separate proprietary innerwarden-watchdog wraps that same supervisor and adds the anti-tamper layer, process stealth, SHA-256 integrity gating (it refuses to respawn a swapped-out binary), and a license check.

Run one or the other. Never run both the supervisor and the standalone agent unit at the same time.

Permissions

The installer creates a dedicated, unprivileged innerwarden service user and grants it only what it needs. You normally never touch this, but here is the shape of it for a manual or hardened setup on Linux:

sudo useradd -r -s /sbin/nologin innerwarden        # dedicated service user
sudo usermod -aG systemd-journal innerwarden        # read the journal without root
sudo usermod -aG docker innerwarden                 # read the Docker socket (container monitoring)
sudo mkdir -p /var/lib/innerwarden
sudo chown innerwarden:innerwarden /var/lib/innerwarden

Response skills that touch the firewall need a narrow, command-scoped sudoers grant for the backend you use. The installer writes the right one; the grant is deliberately limited to the exact command (for example ufw deny from *, or the matching iptables / nft / firewall-cmd rule), never a blanket sudo. The precise drop-in per backend is shown in Trust and Safety Invariants.

Rule that catches people: data_dir in your config must match ReadWritePaths in the systemd unit. If they disagree, the agent cannot write its state and the service flaps.

The kernel-level capabilities (eBPF, raw network capture) are why install needs sudo; the running services keep the minimal set, not full root, except where the watchdog deliberately runs the agent as a root child on hardened production boxes.

Upgrades

Upgrades are toolchain-free and signature-verified, no compiler on the box:

innerwarden upgrade --check               # see if a newer release exists
innerwarden upgrade                        # fetch + verify (SHA-256 / signature) + install the latest

The upgrade keeps your config and data. To check the version you are on at any time:

innerwarden --version

Removing InnerWarden is a single command (innerwarden uninstall, --purge to also drop config and data); the command and its flags are owned by CLI Reference, and Install and First Run links it.

Where things live

Path	What it is
`/etc/innerwarden/`	config (`agent.toml`, `sensor` config, `agent.env` secrets)
`/var/lib/innerwarden/`	data: incidents, decisions, telemetry, baseline, audit trail, models
`/var/log/innerwarden/`	logs (on setups that route there)
`examples/systemd/`	the shipped unit files, including the supervisor unit

All config keys, including data retention, are documented in Configuration; the retention schedule itself and how to export or erase data are owned by Privacy and GDPR.