Monitor Your Server Security with Grafana and Prometheus

InnerWarden exposes a Prometheus-compatible /metrics endpoint on the agent dashboard. This means you can plug it into your existing Grafana + Prometheus stack and get real-time security dashboards alongside your infrastructure monitoring. No additional exporters, no custom scripts.

This tutorial walks through the complete setup: from scraping metrics to building panels that show events by collector, incidents by detector, AI decision latency, and skill executions over time.

What metrics InnerWarden exposes

The /metrics endpoint returns standard Prometheus text format. Here is a sample of what you get:

# HELP innerwarden_events_total Total events processed by collector
# TYPE innerwarden_events_total counter
innerwarden_events_total{collector="auth_log"} 14523
innerwarden_events_total{collector="journald"} 8291
innerwarden_events_total{collector="nginx_access"} 52140
innerwarden_events_total{collector="exec_audit"} 3847
innerwarden_events_total{collector="docker"} 1205

# HELP innerwarden_incidents_total Total incidents by detector
# TYPE innerwarden_incidents_total counter
innerwarden_incidents_total{detector="ssh_bruteforce"} 342
innerwarden_incidents_total{detector="credential_stuffing"} 87
innerwarden_incidents_total{detector="port_scan"} 156
innerwarden_incidents_total{detector="sudo_abuse"} 12
innerwarden_incidents_total{detector="web_scan"} 891

# HELP innerwarden_ai_decision_seconds AI decision latency
# TYPE innerwarden_ai_decision_seconds histogram
innerwarden_ai_decision_seconds_bucket{le="0.5"} 1203
innerwarden_ai_decision_seconds_bucket{le="1.0"} 1287
innerwarden_ai_decision_seconds_bucket{le="2.0"} 1301
innerwarden_ai_decision_seconds_bucket{le="5.0"} 1305
innerwarden_ai_decision_seconds_sum 847.23
innerwarden_ai_decision_seconds_count 1305

# HELP innerwarden_skill_executions_total Skill executions by type
# TYPE innerwarden_skill_executions_total counter
innerwarden_skill_executions_total{skill="block_ip_ufw"} 298
innerwarden_skill_executions_total{skill="block_ip_nftables"} 0
innerwarden_skill_executions_total{skill="monitor_ip"} 45
innerwarden_skill_executions_total{skill="suspend_user_sudo"} 8
innerwarden_skill_executions_total{skill="honeypot_activate"} 14

# HELP innerwarden_blocked_ips_active Currently blocked IPs
# TYPE innerwarden_blocked_ips_active gauge
innerwarden_blocked_ips_active 47

Step 1: Configure Prometheus to scrape InnerWarden

Add the InnerWarden target to your Prometheus configuration. The agent dashboard serves the metrics over HTTPS on 127.0.0.1:8787 by default, with a certificate it generates itself, and asks for the dashboard login unless the scraper runs as root or in the innerwarden group:

scrape_configs:
  - job_name: 'innerwarden'
    scrape_interval: 15s
    scheme: https
    tls_config:
      insecure_skip_verify: true   # the dashboard's own certificate
    basic_auth:
      username: 'your-dashboard-user'
      password_file: /etc/prometheus/innerwarden.pass
    static_configs:
      - targets: ['127.0.0.1:8787']
        labels:
          instance: 'prod-server-01'

The dashboard listens on loopback only, so scrape each server from a Prometheus on that server (or federate them). The instance label lets you filter by server in Grafana.

Verify the scrape is working:

curl -sk -u your-dashboard-user https://127.0.0.1:8787/metrics | head -20

Step 2: Quick setup with Docker Compose

If you do not have Prometheus and Grafana running yet, here is a docker-compose file to get both up quickly:

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    network_mode: host   # serves :9090 on the host, reaches 127.0.0.1:8787
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./innerwarden.pass:/etc/prometheus/innerwarden.pass:ro
      - prometheus_data:/prometheus
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=changeme
    volumes:
      - grafana_data:/var/lib/grafana
    depends_on:
      - prometheus
    restart: unless-stopped

volumes:
  prometheus_data:
  grafana_data:

InnerWarden runs on the host and its dashboard listens on loopback only, so the Prometheus container shares the host's network and keeps 127.0.0.1:8787 as its target. Put the dashboard password in innerwarden.pass next to the compose file: the container does not run as root or in the innerwarden group, so it needs the login.

docker compose up -d

Step 3: Add Prometheus as a Grafana data source

Open Grafana at http://localhost:3000 (default login: admin / changeme). Then:

1. Go to Connections > Data sources > Add data source
2. Select Prometheus
3. Set URL to http://prometheus:9090 (if both run in Docker) or http://localhost:9090 (if Grafana is on the host)
4. Click Save & test. You should see a green success message.

Step 4: Build your security dashboard

Create a new dashboard and add these panels. Each query is copy-paste ready.

rate(innerwarden_events_total[5m])

increase(innerwarden_incidents_total[1h])

histogram_quantile(0.95, rate(innerwarden_ai_decision_seconds_bucket[5m]))

increase(innerwarden_skill_executions_total[1h])

innerwarden_blocked_ips_active

Bonus: Grafana alerts

You can set up Grafana alerts on any of these metrics. Useful alert conditions:

• Incident spike - alert when rate(innerwarden_incidents_total[5m]) > 10 for any detector. This catches sudden attack surges.
• AI latency degradation - alert when P95 latency exceeds 5 seconds. Switch to a faster model or enable the algorithm gate fallback.
• Blocked IP count - alert when active blocks exceed 100. This might indicate a DDoS or a misconfiguration blocking legitimate traffic.

These alerts complement InnerWarden's built-in Telegram and Slack notifications. Grafana gives you the historical trend view. InnerWarden gives you the real-time incident details.

Set it up

Install InnerWarden and the metrics endpoint is available immediately:

npm install -g innerwarden

The eBPF host detection and server-security metrics shown here are InnerWarden Enterprise (Linux); the free innerwarden guardrail screens your agent's commands and MCP/tool calls.

Then point Prometheus at 127.0.0.1:8787/metrics and build your dashboard. The one thing it needs from InnerWarden is the dashboard login.

What to do next

• Telegram alerts - get real-time incident notifications alongside your Grafana dashboards.
• SSH brute-force detection - the most common incident type you will see in your Grafana panels.
• View on GitHub - check the full list of metrics exported by the agent.