12 – Monitoring und Alerting

Revisionshistorie

Inhaltsverzeichnis

1. Cribl-internes Monitoring
2. Externe Health Checks
3. Key Performance Indicators (KPIs)
4. Monitoring-Integration (Prometheus/Grafana)
5. Alerting-Konzept
6. Dashboard-Empfehlungen
7. Cron-basiertes Monitoring Setup

1. Cribl-internes Monitoring

Cribl Stream bietet ein integriertes Monitoring-Dashboard, das ueber die Web-UI unter http://<stream-host>:9000 erreichbar ist. Die folgenden Bereiche werden ueberwacht:

1.1 System Metrics

1.2 Worker Health

1.3 Monitoring-API Endpunkte

# Health Check
curl -s http://localhost:9000/api/v1/health | jq .

# System Info
curl -s -H "Authorization: Bearer ${TOKEN}" \
    http://localhost:9000/api/v1/system/info | jq .

# Worker Status
curl -s -H "Authorization: Bearer ${TOKEN}" \
    http://localhost:9000/api/v1/master/workers | jq .

# Aktive Inputs/Outputs
curl -s -H "Authorization: Bearer ${TOKEN}" \
    http://localhost:9000/api/v1/m/default/system/inputs | jq '.items | length'

# Metriken (internes Metriken-Endpoint)
curl -s -H "Authorization: Bearer ${TOKEN}" \
    http://localhost:9000/api/v1/system/metrics | jq .

2. Externe Health Checks

2.1 Verifikationsskript (07-verify-deployment.sh)

Das Projekt beinhaltet eine umfassende Verifikationssuite unter scripts/07-verify-deployment.sh, die ca. 20 Tests durchfuehrt:

# Verifikationssuite ausfuehren
sudo ./scripts/07-verify-deployment.sh

# Ergebnis: Tabelle mit PASS/FAIL je Test
# Exit-Code: 0 = alle Tests bestanden, 1 = mindestens ein Test fehlgeschlagen

2.2 Einfacher Health Check (curl-basiert)

#!/usr/bin/env bash
# Einfacher Health Check fuer Cron oder externe Monitoring-Systeme

STREAM_URL="http://localhost:9000/api/v1/health"
SPLUNK_HEC="https://10.10.0.66:8088/services/collector/health"

# Cribl Stream Health
stream_status=$(curl -s -o /dev/null -w "%{http_code}" "${STREAM_URL}" 2>/dev/null)
if [[ "${stream_status}" != "200" ]]; then
    echo "CRITICAL: Cribl Stream nicht erreichbar (HTTP ${stream_status})"
    exit 2
fi

# Cribl Edge Service
if ! systemctl is-active cribl-edge &>/dev/null; then
    echo "WARNING: Cribl Edge Service nicht aktiv"
    exit 1
fi

# Splunk HEC
splunk_status=$(curl -sk -o /dev/null -w "%{http_code}" "${SPLUNK_HEC}" 2>/dev/null)
if [[ "${splunk_status}" != "200" ]]; then
    echo "WARNING: Splunk HEC nicht erreichbar (HTTP ${splunk_status})"
    exit 1
fi

echo "OK: Alle Komponenten erreichbar"
exit 0

3. Key Performance Indicators (KPIs)

3.1 KPI-Definitionen

3.2 Datenreduktions-KPIs

4. Monitoring-Integration (Prometheus/Grafana)

4.1 Prometheus-Endpunkt

Cribl Stream kann Metriken im Prometheus-Format bereitstellen. Hierzu wird ein interner Metrics-Endpunkt genutzt oder alternativ ein Exporter konfiguriert.

# Cribl Stream Metriken abrufen (Prometheus-Format)
curl -s -H "Authorization: Bearer ${TOKEN}" \
    http://localhost:9000/api/v1/system/metrics/prometheus

# Alternativ: Eigener Exporter via Script
# Siehe Abschnitt 7 (Cron-basiertes Monitoring)

4.2 Prometheus Scrape-Konfiguration

# prometheus.yml - Scrape-Konfiguration fuer Cribl
scrape_configs:
  - job_name: 'cribl-stream'
    scrape_interval: 30s
    metrics_path: '/api/v1/system/metrics/prometheus'
    bearer_token: '${CRIBL_API_TOKEN}'
    static_configs:
      - targets: ['localhost:9000']
        labels:
          environment: 'poc'
          component: 'cribl-stream'

  - job_name: 'cribl-health'
    scrape_interval: 60s
    metrics_path: '/api/v1/health'
    static_configs:
      - targets: ['localhost:9000']
        labels:
          environment: 'poc'

4.3 Grafana-Integration

Fuer eine visuelle Darstellung der Metriken wird Grafana empfohlen. Konfiguration:

1. Prometheus als Datenquelle in Grafana hinzufuegen
2. Dashboard importieren oder manuell erstellen (siehe Abschnitt 6)
3. Alerting-Regeln in Grafana konfigurieren (als Ergaenzung zu Cribl-eigenen Alerts)

5. Alerting-Konzept

5.1 Schwellwerte und Benachrichtigungen

5.2 Benachrichtigungskanaele

6. Dashboard-Empfehlungen

6.1 Operations Dashboard (Hauptansicht)

Das Operations Dashboard bietet eine Gesamtuebersicht ueber den Zustand der Log-Pipeline:

6.2 Source-Detail Dashboard

6.3 Destination-Detail Dashboard

7. Cron-basiertes Monitoring Setup

7.1 Monitoring-Skript

#!/usr/bin/env bash
# /opt/IceDataEmphasise/scripts/monitoring/health-check.sh
# Cron-basierter Health Check fuer IceDataEmphasise
# Ausfuehrung: alle 5 Minuten via Cron

LOG_FILE="/var/log/cribl-health-check.log"
ALERT_EMAIL="cribl-alerts@bank.example"
TIMESTAMP=$(date '+%Y-%m-%d %H:%M:%S')

STATUS="OK"
MESSAGES=()

# 1. Cribl Stream Health
stream_http=$(curl -s -o /dev/null -w "%{http_code}" \
    http://localhost:9000/api/v1/health 2>/dev/null || echo "000")
if [[ "${stream_http}" != "200" ]]; then
    STATUS="CRITICAL"
    MESSAGES+=("Cribl Stream Health Check fehlgeschlagen (HTTP ${stream_http})")
fi

# 2. Cribl Edge Service
if ! systemctl is-active cribl-edge &>/dev/null; then
    [[ "${STATUS}" != "CRITICAL" ]] && STATUS="WARNING"
    MESSAGES+=("Cribl Edge Service nicht aktiv")
fi

# 3. Splunk HEC Erreichbarkeit
splunk_http=$(curl -sk -o /dev/null -w "%{http_code}" \
    https://10.10.0.66:8088/services/collector/health 2>/dev/null || echo "000")
if [[ "${splunk_http}" != "200" ]]; then
    STATUS="CRITICAL"
    MESSAGES+=("Splunk HEC nicht erreichbar (HTTP ${splunk_http})")
fi

# 4. Festplattenplatz
free_gb=$(df -BG --output=avail /opt 2>/dev/null | tail -1 | tr -d ' G')
if [[ "${free_gb}" -lt 2 ]]; then
    STATUS="CRITICAL"
    MESSAGES+=("Festplattenspeicher kritisch niedrig: ${free_gb} GB frei")
elif [[ "${free_gb}" -lt 5 ]]; then
    [[ "${STATUS}" != "CRITICAL" ]] && STATUS="WARNING"
    MESSAGES+=("Festplattenspeicher niedrig: ${free_gb} GB frei")
fi

# 5. Persistent Queue Groesse
if [[ -d /opt/cribl/state/queues ]]; then
    queue_mb=$(du -sm /opt/cribl/state/queues 2>/dev/null | cut -f1)
    if [[ "${queue_mb}" -gt 2048 ]]; then
        STATUS="CRITICAL"
        MESSAGES+=("Persistent Queue kritisch gross: ${queue_mb} MB")
    elif [[ "${queue_mb}" -gt 512 ]]; then
        [[ "${STATUS}" != "CRITICAL" ]] && STATUS="WARNING"
        MESSAGES+=("Persistent Queue waechst: ${queue_mb} MB")
    fi
fi

# Ergebnis loggen
if [[ ${#MESSAGES[@]} -eq 0 ]]; then
    echo "${TIMESTAMP} [${STATUS}] Alle Checks bestanden" >> "${LOG_FILE}"
else
    for msg in "${MESSAGES[@]}"; do
        echo "${TIMESTAMP} [${STATUS}] ${msg}" >> "${LOG_FILE}"
    done
fi

# Bei WARNING oder CRITICAL: E-Mail senden
if [[ "${STATUS}" != "OK" ]]; then
    {
        echo "IceDataEmphasise Health Check - ${STATUS}"
        echo "Zeitpunkt: ${TIMESTAMP}"
        echo ""
        for msg in "${MESSAGES[@]}"; do
            echo "- ${msg}"
        done
    } | mail -s "[${STATUS}] IceDataEmphasise Health Check" "${ALERT_EMAIL}"
fi

exit 0

7.2 Cron-Konfiguration

# Crontab-Eintraege (sudo crontab -e)

# Health Check alle 5 Minuten
*/5 * * * * /opt/IceDataEmphasise/scripts/monitoring/health-check.sh

# Vollstaendige Verifikation taeglich um 06:00
0 6 * * * /opt/IceDataEmphasise/scripts/07-verify-deployment.sh \
    >> /var/log/cribl-verify.log 2>&1

# Log-Rotation woechentlich
0 0 * * 0 find /var/log/cribl-health-check.log -size +10M \
    -exec mv {} {}.old \; 2>/dev/null

7.3 Log-Rotation

# /etc/logrotate.d/cribl-monitoring
/var/log/cribl-health-check.log
/var/log/cribl-verify.log {
    weekly
    rotate 12
    compress
    delaycompress
    missingok
    notifempty
    create 0640 root root
}