False Positives Are a Feature Problem, Not a Tuning Problem

I woke up to 130 Telegram alerts from a production server. One hundred and thirty. Overnight. My phone had been buzzing for hours. Any reasonable person would have muted the channel by the third notification, and that is exactly the problem. When your security alerts become noise, you stop reading them. Then the real attack arrives and you miss it.

Here is the breakdown of those 130 alerts: 49 were ssh_bruteforce (real attacks, legitimate detections), 42 were Sigma rule matches (false positives from generic rules), 22 were sensitive_write detections (false positives from normal system activity), and 18 were data_exfil alerts (false positives from a compiler running build scripts). Only the SSH brute force alerts were real. The rest were the monitoring system screaming about perfectly normal operations.

Why false positives happen in eBPF monitoring

When you monitor at the kernel level with eBPF, you see everything. Every syscall, every file open, every network connection, every process spawn. This is the entire point of eBPF security. But "everything" includes the operating system doing its job. Package managers reading system files. Build tools spawning shell processes. DNS resolvers making outbound connections. Monitoring agents reading configuration.

The same syscall pattern that indicates data exfiltration also happens when GCC compiles a project. The same file access that signals credential theft also happens when PAM authenticates a legitimate user. The same process execution chain that reveals a reverse shell also happens when npm runs a build script.

Traditional security tools solve this by reducing visibility. They only watch logs, or only monitor specific directories, or only track known-bad signatures. eBPF gives you full visibility. The challenge is teaching the system what "normal" looks like at that level of detail.

Six false positives we actually hit (and how we fixed each one)

These are not hypothetical. Every example below came from a real production alert that woke someone up or clogged a Telegram channel. Each one required understanding the root cause, not adjusting a threshold.

// Before: flagged because argv contained system paths
// cc -I/usr/include -L/usr/lib/gcc/ main.c -o main

// Fix: check the actual binary, not just argv contents
fn is_build_tool(cmd: &str) -> bool {
    let binary = cmd.split_whitespace()
        .next()
        .unwrap_or("")
        .rsplit('/')
        .next()
        .unwrap_or("");
    matches!(binary,
        "cc" | "gcc" | "g++" | "clang" | "clang++"
        | "rustc" | "cargo" | "make" | "cmake"
        | "ld" | "as" | "ar"
    )
}

// The detection sees:
//   /bin/sh -c react-scripts build
//   parent: node (/usr/bin/node)
//
// Looks like: webshell spawning a shell
// Actually is: npm running a build script

fn is_build_command(argv: &str) -> bool {
    let after_sh = argv.strip_prefix("/bin/sh -c ")
        .or_else(|| argv.strip_prefix("/bin/bash -c "));
    if let Some(cmd) = after_sh {
        let tool = cmd.split_whitespace()
            .next()
            .unwrap_or("");
        return matches!(tool,
            "react-scripts" | "next" | "vite"
            | "webpack" | "tsc" | "eslint"
            | "prettier" | "jest" | "vitest"
            | "npx" | "yarn" | "pnpm"
        );
    }
    false
}

// CrowdSec cscli makes direct DNS queries
// Detector sees: unknown process -> 1.1.1.1:53
// Looks like: DNS exfiltration tunnel
// Actually is: CrowdSec updating its blocklists

// Static allowlist in the detector code
const INFRA_DNS_PROCESSES: &[&str] = &[
    "cscli",
    "crowdsec",
    "systemd-resolved",
    "unbound",
];

fn is_allowed_dns_query(process: &str) -> bool {
    let binary = process.rsplit('/').next().unwrap_or("");
    INFRA_DNS_PROCESSES.contains(&binary)
}

// InnerWarden's own process reading PAM config
// Process: tokio-rt-worker (tid of innerwarden agent)
// Path: /etc/pam.d/common-auth
// Detection: sensitive_write

// Fix: know thyself
fn is_own_process(uid: u32) -> bool {
    // InnerWarden runs as a dedicated service user
    // Skip all file access events from our own UID
    uid == INNERWARDEN_UID
}

// Applied early in the detection pipeline
if is_own_process(event.uid) {
    return None; // no incident
}

// Admin deploy: systemctl stop innerwarden
// Detection: service_tamper (stopping security service)
// This IS what attackers do. But also what admins do.

fn is_legitimate_service_management(
    uid: u32,
    argv: &str,
) -> bool {
    // Root user managing innerwarden's own service
    if uid == 0 {
        let args: Vec<&str> = argv.split_whitespace().collect();
        if let Some(svc) = args.last() {
            return svc.starts_with("innerwarden");
        }
    }
    false
}

// Still logs the event, just doesn't alert
// Audit trail is preserved, Telegram stays quiet

# Sigma rule that fires on everything
# Rule ID: d8c55a9b-... "Inline Python Execution"
# Matched: /bin/sh -c ip neigh show
# 42 alerts overnight from this single rule

# Fix: suppress specific rule IDs in config
# innerwarden.toml
[allowlist.sigma]
suppress_rule_ids = [
    "d8c55a9b-...",  # Inline Python - too broad for servers
    "a1b2c3d4-...",  # Generic shell spawn - useless with eBPF
]

The systematic approach: static + dynamic allowlists

After hitting enough false positives, a pattern emerged. Some fixes belong in code (they are always true, on every server) and some belong in configuration (they depend on what software the server runs). Mixing these up causes problems. Putting environment-specific fixes in code means rebuilding for every new deployment. Putting universal fixes in config means every user has to rediscover the same solutions.

Inner Warden uses two layers:

• Static allowlists (in code): universal truths that apply everywhere. Build tools are not exfiltration. Your own process reading your own config is not suspicious. Root managing its own service is not tampering. These ship with the binary.
• Dynamic allowlists (in config): environment-specific suppressions. CrowdSec runs on this server. This IP range is the monitoring subnet. This Sigma rule is too noisy for this workload. These live in innerwarden.toml and require no rebuild.

The dynamic allowlist format

The TOML configuration gives you fine-grained control over what gets suppressed. You can allowlist by process name, IP address, CIDR range, port, or specific detector. Every suppression is logged in the audit trail so you can review what was silenced.

[allowlist]
# Processes that should never trigger alerts
processes = [
    "cscli",
    "crowdsec",
    "promtail",
    "node_exporter",
]

# Trusted IP addresses (monitoring, CI/CD, etc.)
ips = ["10.0.0.50", "10.0.0.51"]

# Trusted subnets
cidrs = ["10.0.0.0/24", "172.16.0.0/12"]

# Ports to ignore for outbound connection alerts
ports = [443, 8443, 9090]  # HTTPS, Prometheus

# Per-detector suppressions
[allowlist.detectors]
data_exfil = { skip_binaries = ["cc", "gcc", "cargo", "npm"] }
sensitive_write = { skip_paths = ["/etc/letsencrypt/"] }
service_tamper = { skip_services = ["innerwarden"] }

# Sigma rule suppression
[allowlist.sigma]
suppress_rule_ids = [
    "d8c55a9b-...",  # Inline Python Execution
]

This file is hot-reloaded. Edit it, and the changes take effect within seconds. No restart, no rebuild, no downtime. When you discover a new false positive at 3 AM, you add one line to the config and go back to sleep.

Telegram batching: stop the flood

Even after fixing false positives, legitimate alerts can flood your phone. An SSH brute force attack generates dozens of events per minute. You need to know it is happening, but you do not need 49 individual notifications. Inner Warden uses a batching strategy for Telegram alerts.

• First occurrence: sent immediately. You know about it within seconds.
• Repeated events: grouped into 60-second summaries. Instead of 49 messages saying "SSH brute force from 185.x.x.x," you get one message: "SSH brute force: 49 attempts from 12 IPs in the last 60 seconds. Top attacker: 185.x.x.x (23 attempts)."

[telegram]
token = "your-bot-token"
chat_id = "your-chat-id"

# First alert: immediate
# Subsequent alerts of the same type: batched
batch_window_secs = 60

# Summary format includes:
# - total count
# - unique source IPs
# - top attacker
# - detector name
# - time window

The real lesson: FPs are not solved by tuning thresholds

The instinct when you get false positives is to adjust thresholds. Raise the connection count from 5 to 10. Increase the time window from 60 to 300 seconds. Require 3 matching conditions instead of 2. This approach fails because it reduces your ability to detect real attacks. Every threshold increase is a gift to attackers who stay just below it.

The six examples above share a common pattern: none of them were fixed by changing a number. Every fix required understanding what specific behavior caused the false positive and encoding that knowledge into the detector. GCC is a build tool, not an exfiltration vector. npm spawns shells by design, not by compromise. Your own process reading your own config is expected, not suspicious.

False positives are a feature problem. They mean the detector does not understand the context well enough. The fix is always the same: figure out what "normal" looks like for that specific pattern, and teach the detector to recognize it.

1. Alert fires on legitimate behavior
2. Identify the root cause (not the symptom)
3. Ask: is this always true, or environment-specific?
4. Always true   -> fix in code (static allowlist)
   Env-specific  -> fix in config (dynamic allowlist)
5. Preserve audit trail (log suppressed events)
6. Never raise thresholds to hide the problem

Before and after

After applying these fixes to the production server that generated 130 alerts overnight:

• Before: 130 Telegram messages overnight. 62% false positive rate. Alert fatigue. Real incidents buried in noise.
• After: 3 Telegram messages overnight. 49 brute force attempts compressed into one batched summary. 2 real anomalies that deserved attention. Zero false positives.

The detection capability did not decrease. Every eBPF hook still fires. Every event still gets evaluated. The difference is that the system now understands what normal looks like on this server. The suppressed events are still logged in the JSONL audit trail. They just do not wake you up at 3 AM.

What to do next

• eBPF for Security to understand the six eBPF programs that generate these events in the first place.
• Telegram security alerts to set up batched alerting with deduplication and severity-based routing.
• Baseline learning to see how automated baseline profiling reduces false positives by learning what is normal during a 7-day training window.