Skip to content
All posts
Monitoring

Monitor Your Server Security with Grafana and Prometheus

10 min read

Inner Warden exposes a Prometheus-compatible /metrics endpoint on the agent dashboard. This means you can plug it into your existing Grafana + Prometheus stack and get real-time security dashboards alongside your infrastructure monitoring. No additional exporters, no custom scripts.

This tutorial walks through the complete setup: from scraping metrics to building panels that show events by collector, incidents by detector, AI decision latency, and skill executions over time.

What metrics Inner Warden exposes

The /metrics endpoint returns standard Prometheus text format. Here is a sample of what you get:

Sample /metrics output
# HELP innerwarden_events_total Total events processed by collector
# TYPE innerwarden_events_total counter
innerwarden_events_total{collector="auth_log"} 14523
innerwarden_events_total{collector="journald"} 8291
innerwarden_events_total{collector="nginx_access"} 52140
innerwarden_events_total{collector="exec_audit"} 3847
innerwarden_events_total{collector="docker"} 1205

# HELP innerwarden_incidents_total Total incidents by detector
# TYPE innerwarden_incidents_total counter
innerwarden_incidents_total{detector="ssh_bruteforce"} 342
innerwarden_incidents_total{detector="credential_stuffing"} 87
innerwarden_incidents_total{detector="port_scan"} 156
innerwarden_incidents_total{detector="sudo_abuse"} 12
innerwarden_incidents_total{detector="web_scan"} 891

# HELP innerwarden_ai_decision_seconds AI decision latency
# TYPE innerwarden_ai_decision_seconds histogram
innerwarden_ai_decision_seconds_bucket{le="0.5"} 1203
innerwarden_ai_decision_seconds_bucket{le="1.0"} 1287
innerwarden_ai_decision_seconds_bucket{le="2.0"} 1301
innerwarden_ai_decision_seconds_bucket{le="5.0"} 1305
innerwarden_ai_decision_seconds_sum 847.23
innerwarden_ai_decision_seconds_count 1305

# HELP innerwarden_skill_executions_total Skill executions by type
# TYPE innerwarden_skill_executions_total counter
innerwarden_skill_executions_total{skill="block_ip_ufw"} 298
innerwarden_skill_executions_total{skill="block_ip_nftables"} 0
innerwarden_skill_executions_total{skill="monitor_ip"} 45
innerwarden_skill_executions_total{skill="suspend_user_sudo"} 8
innerwarden_skill_executions_total{skill="honeypot_activate"} 14

# HELP innerwarden_blocked_ips_active Currently blocked IPs
# TYPE innerwarden_blocked_ips_active gauge
innerwarden_blocked_ips_active 47

Step 1: Configure Prometheus to scrape Inner Warden

Add the Inner Warden target to your Prometheus configuration. The agent dashboard runs on port 9111 by default:

prometheus.yml
scrape_configs:
  - job_name: 'innerwarden'
    scrape_interval: 15s
    static_configs:
      - targets: ['localhost:9111']
        labels:
          instance: 'prod-server-01'

If you run multiple servers with Inner Warden, add each one as a separate target. The instance label lets you filter by server in Grafana.

Verify the scrape is working:

curl -s http://localhost:9111/metrics | head -20

Step 2: Quick setup with Docker Compose

If you do not have Prometheus and Grafana running yet, here is a docker-compose file to get both up quickly:

docker-compose.yml
version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=changeme
    volumes:
      - grafana_data:/var/lib/grafana
    depends_on:
      - prometheus
    restart: unless-stopped

volumes:
  prometheus_data:
  grafana_data:

Since Inner Warden runs on the host (not in Docker), use host.docker.internal:9111 as the target in your prometheus.yml when Prometheus runs in Docker.

docker compose up -d

Step 3: Add Prometheus as a Grafana data source

Open Grafana at http://localhost:3000 (default login: admin / changeme). Then:

  1. Go to Connections > Data sources > Add data source
  2. Select Prometheus
  3. Set URL to http://prometheus:9090 (if both run in Docker) or http://localhost:9090 (if Grafana is on the host)
  4. Click Save & test. You should see a green success message.

Step 4: Build your security dashboard

Create a new dashboard and add these panels. Each query is copy-paste ready.

Panel 1: Events by collector (time series)
rate(innerwarden_events_total[5m])

Shows events per second for each collector. Use Legend {{collector}} for readable labels.

Panel 2: Incidents by detector (bar chart)
increase(innerwarden_incidents_total[1h])

Hourly incident count per detector. Useful for identifying which attack types are most active.

Panel 3: AI decision latency (histogram)
histogram_quantile(0.95, rate(innerwarden_ai_decision_seconds_bucket[5m]))

P95 AI decision time. If this exceeds 3 seconds, your AI provider may be throttling or the model is too large.

Panel 4: Skill executions (stacked area)
increase(innerwarden_skill_executions_total[1h])

Hourly breakdown of automated actions. Shows block-ip, suspend-user, honeypot activations, and monitoring actions.

Panel 5: Active blocked IPs (stat panel)
innerwarden_blocked_ips_active

Current count of blocked IPs. This gauge goes up when attacks are blocked and down when TTLs expire.

Bonus: Grafana alerts

You can set up Grafana alerts on any of these metrics. Useful alert conditions:

  • Incident spike - alert when rate(innerwarden_incidents_total[5m]) > 10 for any detector. This catches sudden attack surges.
  • AI latency degradation - alert when P95 latency exceeds 5 seconds. Switch to a faster model or enable the algorithm gate fallback.
  • Blocked IP count - alert when active blocks exceed 100. This might indicate a DDoS or a misconfiguration blocking legitimate traffic.

These alerts complement Inner Warden's built-in Telegram and Slack notifications. Grafana gives you the historical trend view. Inner Warden gives you the real-time incident details.

Set it up

Install Inner Warden and the metrics endpoint is available immediately:

Install
curl -fsSL https://innerwarden.com/install | sudo bash

Then point Prometheus at localhost:9111/metrics and build your dashboard. No extra configuration needed on the Inner Warden side.

What to do next