AI-generated content

A good monitoring setup is the difference between “the network is down” and “I noticed packet loss on eth0 at 2:14 AM and already sent a fix.”

Architecture

[Devices] → [Exporters] → [Prometheus] → [Grafana]
                    ↓
                 [Alertmanager] → [Notifications]
Component Purpose
Prometheus Time-series database, scraping, alerting
Grafana Visualization, dashboards, alerting
Alertmanager Deduplication, routing, notifications
Exporters System/network metrics collection

Components to Monitor

Servers

  • node_exporter: CPU, memory, disk, network, temperatures
  • cAdvisor: Container metrics (memory, CPU, filesystem)
  • blackbox_exporter: HTTP/TCP/ICMP probing

Network

  • snmp_exporter: Router/switch metrics via SNMP
  • nfdump_exporter: NetFlow data for traffic analysis
  • Suricata: IDS/IPS metrics

Services

  • Prometheus itself: Scrape health
  • Application exporters: Custom app metrics
  • Uptime Kuma: Simple uptime monitoring

Quick Start

Docker Compose Setup

version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=30d'
      - '--storage.tsdb.path=/prometheus'
    ports:
      - "9090:9090"
    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=your-strong-password
    ports:
      - "3000:3000"
    depends_on:
      - prometheus
    restart: unless-stopped

  alertmanager:
    image: prom/alertmanager:latest
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    ports:
      - "9093:9093"
    restart: unless-stopped

volumes:
  prometheus-data:
  grafana-data:

Prometheus Configuration

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - "alert.rules"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['node_exporter:9100']

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
        - https://example.com
        - https://github.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_exporter:9115

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

Essential Dashboards

1. System Overview (node_exporter)

  • CPU utilization by core
  • Memory pressure
  • Disk I/O and space
  • Network throughput
  • System load averages

2. Network Traffic

  • Bandwidth per interface
  • Top talkers (with nfdump)
  • Error rates
  • Packet drops
  • Connection tracking

3. Service Health

  • Uptime per service
  • Response times
  • Error rates
  • Certificate expiry
  • DNS resolution times

4. Security

  • Failed SSH attempts (fail2ban metrics)
  • Port scan detection
  • Firewall rule hits
  • File integrity changes

Alert Rules

groups:
  - name: infrastructure
    rules:
      - alert: HighCPU
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU on {{ $labels.instance }}"
          description: "{{ $labels.instance }} CPU usage > 90% for 5 minutes"

      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 < 10
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"

      - alert: ServiceDown
        expr: up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Service {{ $labels.job }} is down"

  - name: network
    rules:
      - alert: HighNetworkErrors
        expr: rate(node_network_receive_errs_total[5m]) + rate(node_network_transmit_errs_total[5m]) > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Network errors on {{ $labels.instance }}"

Notification Channels

Email (SMTP)

# alertmanager.yml
route:
  receiver: 'email'
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m

receivers:
  - name: 'email'
    email_configs:
      - to: '[email protected]'
        from: '[email protected]'
        smarthost: 'smtp.example.com:587'

Telegram Bot

  - name: 'telegram'
    telegram_configs:
      - bot_token: 'YOUR_BOT_TOKEN'
        chat_id: YOUR_CHAT_ID
        message_format: 'html'

Discord Webhook

  - name: 'discord'
    discord_configs:
      - webhook_url: 'https://discord.com/api/webhooks/...'
        send_resolved: true

Advanced: NetFlow Analysis

For deeper network visibility, collect NetFlow/sFlow data:

# Install nfdump
sudo apt install nfdump

# Configure your router to send NetFlow to your server
# Then export to Prometheus
nfcapd -p /var/nfdump -l -T -L 4h

Maintenance

Task Frequency
Check Prometheus health Daily
Verify alerts firing Weekly
Review disk usage Weekly
Update dashboards As needed
Rotate Grafana passwords Monthly
Review alert noise Weekly

Budget

Component Cost
All software Free (open source)
Raspberry Pi 4 (2GB) $50
USB drive (32GB) $10
Total ~$60

The Raspberry Pi can comfortably handle Prometheus + Grafana + a handful of exporters for a home network.

This post was generated with AI assistance. Review and customize it to match your experience.