Operate

Responding to Incidents

Name: InnerWarden
Author: InnerWarden

How to read an InnerWarden incident, the response actions available, how automatic containment works, and the guardrails you can trust.

Responding to Incidents

For: operators handling a live alert on a host InnerWarden is protecting.

An incident is InnerWarden telling you "I saw something on this machine that looks like an attack, here is what it is and here is what I did or want to do about it." This page is how to read one, what your options are, and how automatic response works so you trust what it does on its own.

If you are tuning out noise rather than handling a real attack, that belongs in Everyday Operations (quieting) or Safe Observe and Allowlist (trusting a specific benign source). This page is for when something might actually be wrong.

Read the incident first

Start by looking at what fired:

innerwarden get incidents --days 7
innerwarden get incidents --days 1 --severity high

Every incident carries a plain-language explained alert: a "what happened and why it matters" line plus the MITRE ATT&CK technique it maps to. You read it as understanding ("InnerWarden saw this, knows what it is, and is handling it"), not as a raw detector name. That explanation comes through on the dashboard and in every notification too, so you get the same story on your phone as in the terminal.

To pivot from an incident to the full story of one attacker:

innerwarden get entity <ip-or-user>       # full timeline: every event, decision, and enrichment for that entity

The entity view stitches together everything InnerWarden knows about that IP or user: when it first appeared, what it tried, its reputation (AbuseIPDB, DShield, GeoIP), and how it was handled. A paid host's dashboard tells the same story as a case, filed under one of three lanes and read as what happened, what InnerWarden did and what you need to decide (from 0.16.67; on 0.16.66 and older it is a case in the Cases list). See Dashboard and Notifications.

To see what the agent decided and whether the action actually landed:

innerwarden get decisions --days 7
innerwarden get responses                 # the outcome of each response action

Your response options

Whether the agent acted automatically or left it to you, these are the responses available. The most common manual ones are simple commands:

innerwarden action block <ip> --reason "<why>"      # block an attacking IP
sudo innerwarden-ctl action unblock <ip> --reason "<why>"    # reverse a block (false positive, or you are done)

The full set of responses InnerWarden can take, automatically or on your say-so:

Response	What it does
Block IP	Drops the attacker's traffic at the firewall. Five Linux backends (ufw, iptables, nftables, firewalld, and wire-speed XDP), auto-selected for your host. XDP drops only what the address sends in; the other four can block both ways.
Unblock IP	Reverses a block through the response lifecycle. The one path the reconciler will not silently re-apply.
Kill process	Stops the process a confirmed shell names and its direct children, then reads back from the kernel that they are gone. Needs the agent to run as root under the watchdog (the paid install's layout); a process it cannot stop is raised to you as a High finding rather than reported as stopped.
Suspend user	Drops a sudoers deny file with a TTL, auto-cleans on expiry.
Block container	Pauses a Docker container, auto-unpauses after its TTL.
Rate-limit (nginx)	An HTTP 403 deny at the web layer, with TTL and auto-cleanup.
Monitor	Captures limited traffic for the IP (tcpdump plus metadata) without blocking, so you can watch instead of act.
Honeypot redirect	Sends the attacker into an SSH/HTTP decoy with a fake shell, captures their credentials, commands, and the IOCs they reveal (C2 servers, malware drop points). The decoy's shell records commands and answers them with text rather than running them; when a model plays that shell (`llm_shell`), it only writes the decoy's text. Whether you are asked first or it just goes is `[honeypot] on_ai_decision` (`ask` or `automatic`); on an `always_on` host the redirect goes into the listener that is already running.

Every response is bounded, audited, and reversible. None of them is fire-and-forget: each has a TTL where that makes sense, each writes to the audit trail, and each can be undone. In dry-run mode (the default at install) they log exactly what they would do without doing it. The exact flags for each command live in CLI Reference.

You can also act from your phone (Telegram inline buttons on an alert: Block the address, Ignore, Not a threat (Report FP on the technical profile), plus /unblock <ip>) or from the dashboard. Both surfaces are covered in Dashboard and Notifications.

Automatic vs. manual response

By default, a fresh install is observe-only and dry-run: it detects and explains, it does not block or kill anything until you arm it. That is intentional, you decide when it is allowed to act. Arming enforcement safely (after a clean baseline and a verified trust list) is the workflow in Safe Observe and Allowlist.

Once armed, the agent can respond on its own for clear-cut cases. The decision path runs each incident through enrichment (IP reputation, GeoIP), then an optional AI provider or the on-device Local Warden model, then a confidence and threshold check, before any skill fires. The detail of that pipeline is in Architecture; what matters as an operator is the guardrails around it:

Ambiguous alerts wait for your decision. When an incident is not clearly an attack and not clearly benign, InnerWarden does not guess and it does not silently drop it. It marks it as needing review, and it waits for you in Cases on the dashboard. A Low or Medium one nobody decides closes after 24 hours with "timed out" as its recorded reason, never as a verdict; a High or Critical one never closes on its own. A review item does not page you by itself: what reaches your phone is what passes the notification gate in Dashboard and Notifications, and a Telegram message for every review item is opt-in ([learning] needs_review_notify = true).
A second opinion, never the only gate. When an AI provider is configured, it can ask for human review rather than auto-acting. Its role is advisory. When no AI is configured, the full review path still runs deterministically, the AI is an enhancement, never a dependency.
Your verdicts teach it. Not a threat (Report FP on the technical profile) is recorded as a label for the Local Warden model, so it gets better at your host over time. Ignore only silences that message: it records no verdict, so a real attack you ignore is never learned as harmless.

When the kernel acts instantly

For the most dangerous patterns, the eBPF layer can refuse an exec in the kernel before the agent even triages, when that guard is in enforce (in observe it records what it would have refused). When the kill-chain detection in the kernel refuses an exec, the response pipeline follows: the sensor raises a Critical incident, the agent stops the process and its direct children (when it runs as root under the watchdog), blocks the C2 address through the configured firewall backend (XDP alone only drops what the address sends in), captures forensics (a /proc snapshot plus socket state), propagates the block to any mesh peers, and notifies you. This is the one path where containment happens first and triage follows, because the alternative is letting the exploit run.

High and Critical incidents also trigger automatic forensics capture (a packet capture plus a /proc snapshot) so the evidence exists even if the attacker cleans up.

The circuit breaker

InnerWarden will not let an alert storm turn into a response storm. Two limits protect both you and the box:

AI rate limiting and a circuit breaker. The agent caps AI calls per tick and, when incident volume spikes, suspends AI triage for that tick and auto-resets after a cooldown. A flood cannot run up your AI bill or stampede the decision path.
Decision cooldown. The same action against the same detector and entity will not re-fire within a cooldown window, so one persistent attacker does not generate a thousand duplicate blocks.
Burst summary. When 50 or more auto-blocks happen in an hour, you get a single "heavy attack, all contained" message that names the server and breaks down what kind of attack it was, instead of 50 separate alerts. That is normal internet background noise being handled, not 50 emergencies.

Containment you can trust

A block is only worth something if it is real. InnerWarden verifies containment against the live firewall, not an internal record:

A High or Critical case whose IP the firewall is actually dropping reads as Contained, verified by checking the live ufw/iptables/nftables state, not by trusting a row in a database. If the firewall and the record disagree, the live firewall wins and the block is re-applied. An attacker cannot get a free pass because some internal record claimed it was already blocked.
This truthful-containment behavior is scoped to recon-class signals, so anything showing active harm stays surfaced in "needs your attention" even when its IP is blocked. You never lose sight of an attacker that is still doing damage just because their traffic is being dropped.

When you reverse a containment, use sudo innerwarden-ctl action unblock (or /unblock <ip> from Telegram). From 0.16.68 the command asks the agent to revert the block, so the agent's own record lets go of it and the reconciler does not put it back, and it prints what the agent did. The agent takes it only from root on this host, outside any guarded AI agent. With the agent stopped, the command removes the rule itself and says plainly that the agent will put it back when it starts: start the agent and run it again.

The audit trail

Every decision and every action is written to a hash-chained local audit trail on the box. It records what was detected, what the agent (or you) decided, the confidence, the target, and the result, chained so tampering is detectable. It is yours and it stays local, there is no cloud control plane.

You can verify the chain and review admin actions through the agent's Compliance endpoint. How long each record is kept (and how to export or erase it) is owned by Privacy and GDPR; how the audit trail maps to compliance controls is in ISO 27001 Mapping.