🈂️ 👩🏿‍🤝‍👩🏽 👩🏿‍💼 Prometheus: monitoreo HTTP a través del exportador Blackbox ✡️ 📕 🍃

Hola a todos. En mayo, OTUS lanza un taller sobre monitoreo y registro , tanto de infraestructura como de aplicaciones, usando Zabbix, Prometheus, Grafana y ELK. En este sentido, tradicionalmente compartimos material útil sobre el tema.

El exportador de Blackbox para Prometheus le permite monitorear servicios externos a través de HTTP, HTTPS, DNS, TCP, ICMP. En este artículo, le mostraré cómo configurar el monitoreo HTTP / HTTPS utilizando el exportador Blackbox. Lanzaremos el exportador de Blackbox en Kubernetes.

Medio ambiente

Necesitaremos lo siguiente:

Kubernetes
Operador Prometeo

Configuración del exportador de Blackbox

Configuramos Blackbox ConfigMappara configurar httpel módulo de monitoreo de servicios web.

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app: prometheus-blackbox-exporter
data:
  blackbox.yaml: |
    modules:
      http_2xx:
        http:
          no_follow_redirects: false
          preferred_ip_protocol: ip4
          valid_http_versions:
          - HTTP/1.1
          - HTTP/2
          valid_status_codes: []
        prober: http
        timeout: 5s

El módulo se http_2xxutiliza para verificar que el servicio web devuelve un código de estado HTTP 2xx. La configuración del exportador de blackbox se describe con más detalle en la documentación .

Implementar exportador de blackbox en el clúster de Kubernetes

Describa Deploymenty Servicedespliegue en Kubernetes.

---
kind: Service
apiVersion: v1
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app: prometheus-blackbox-exporter
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 9115
      protocol: TCP
  selector:
    app: prometheus-blackbox-exporter

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-blackbox-exporter
  labels:
    app: prometheus-blackbox-exporter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-blackbox-exporter
  template:
    metadata:
      labels:
        app: prometheus-blackbox-exporter
    spec:
      restartPolicy: Always
      containers:
        - name: blackbox-exporter
          image: "prom/blackbox-exporter:v0.15.1"
          imagePullPolicy: IfNotPresent
          securityContext:
            readOnlyRootFilesystem: true
            runAsNonRoot: true
            runAsUser: 1000
          args:
            - "--config.file=/config/blackbox.yaml"
          resources:
            {}
          ports:
            - containerPort: 9115
              name: http
          livenessProbe:
            httpGet:
              path: /health
              port: http
          readinessProbe:
            httpGet:
              path: /health
              port: http
          volumeMounts:
            - mountPath: /config
              name: config
        - name: configmap-reload
          image: "jimmidyson/configmap-reload:v0.2.2"
          imagePullPolicy: "IfNotPresent"
          securityContext:
            runAsNonRoot: true
            runAsUser: 65534
          args:
            - --volume-dir=/etc/config
            - --webhook-url=http://localhost:9115/-/reload
          resources:
            {}
          volumeMounts:
            - mountPath: /etc/config
              name: config
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: prometheus-blackbox-exporter

El exportador de blackbox se puede implementar con el siguiente comando. El espacio de nombres se monitoringrefiere al operador Prometheus.

kubectl --namespace=monitoring apply -f blackbox-exporter.yaml

Verifique que todos los servicios se estén ejecutando con el siguiente comando:

kubectl --namespace=monitoring get all --selector=app=prometheus-blackbox-exporter

Comprobar caja negra

Puede acceder a la interfaz web de Blackbox del exportador con port-forward:

kubectl --namespace=monitoring port-forward svc/prometheus-blackbox-exporter 9115:9115

Conéctese a la interfaz web Blackbox del exportador a través de un navegador web en localhost : 9115.

Si va a la dirección http: // localhost: 9115 / probe? Module = http_2xx & target = https: //www.google.com , verá el resultado de verificar la URL especificada ( https://www.google.com ).

Un probe_successvalor métrico de 1 significa una verificación exitosa. Un valor de 0 indica un error.

Configurar Prometeo

Después de implementar el exportador BlackBox, configure Prometheus prometheus-additional.yaml.

- job_name: 'kube-api-blackbox'
  scrape_interval: 1w
  metrics_path: /probe
  params:
    module: [http_2xx]
  static_configs:
   - targets:
      - https://www.google.com
      - http://www.example.com
      - https://prometheus.io
  relabel_configs:
   - source_labels: [__address__]
     target_label: __param_target
   - source_labels: [__param_target]
     target_label: instance
   - target_label: __address__
     replacement: prometheus-blackbox-exporter:9115 # The blackbox exporter.

Genere Secretusando el siguiente comando.

PROMETHEUS_ADD_CONFIG=$(cat prometheus-additional.yaml | base64)
cat << EOF | kubectl --namespace=monitoring apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: additional-scrape-configs
type: Opaque
data:
  prometheus-additional.yaml: $PROMETHEUS_ADD_CONFIG
EOF

Especifique additional-scrape-configspara el Operador Prometheus usando additionalScrapeConfigs.

kubectl --namespace=monitoring edit prometheuses k8s
...
spec:
  additionalScrapeConfigs:
    key: prometheus-additional.yaml
    name: additional-scrape-configs

Entramos en la interfaz web de Prometheus, verificamos las métricas y los objetivos.

kubectl --namespace=monitoring port-forward svc/prometheus-k8s 9090:9090

Vemos las métricas y los objetivos de Blackbox.

Agregar reglas para notificaciones (alerta)

Para recibir notificaciones del exportador de Blackbox, agregue las reglas al Operador Prometheus.

kubectl --namespace=monitoring edit prometheusrules prometheus-k8s-rules
...
  - name: blackbox-exporter
    rules:
    - alert: ProbeFailed
      expr: probe_success == 0
      for: 5m
      labels:
        severity: error
      annotations:
        summary: "Probe failed (instance {{ $labels.instance }})"
        description: "Probe failed\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"
    - alert: SlowProbe
      expr: avg_over_time(probe_duration_seconds[1m]) > 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Slow probe (instance {{ $labels.instance }})"
        description: "Blackbox probe took more than 1s to complete\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"
    - alert: HttpStatusCode
      expr: probe_http_status_code <= 199 OR probe_http_status_code >= 400
      for: 5m
      labels:
        severity: error
      annotations:
        summary: "HTTP Status Code (instance {{ $labels.instance }})"
        description: "HTTP status code is not 200-399\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"
    - alert: SslCertificateWillExpireSoon
      expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 30
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "SSL certificate will expire soon (instance {{ $labels.instance }})"
        description: "SSL certificate expires in 30 days\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"
    - alert: SslCertificateHasExpired
      expr: probe_ssl_earliest_cert_expiry - time()  <= 0
      for: 5m
      labels:
        severity: error
      annotations:
        summary: "SSL certificate has expired (instance {{ $labels.instance }})"
        description: "SSL certificate has expired already\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"
    - alert: HttpSlowRequests
      expr: avg_over_time(probe_http_duration_seconds[1m]) > 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "HTTP slow requests (instance {{ $labels.instance }})"
        description: "HTTP request took more than 1s\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"
    - alert: SlowPing
      expr: avg_over_time(probe_icmp_duration_seconds[1m]) > 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Slow ping (instance {{ $labels.instance }})"
        description: "Blackbox ping took more than 1s\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

En la interfaz web de Prometheus, vaya a Estado => Reglas y busque las reglas de alerta para blackbox-exporter.

Configurar las notificaciones de vencimiento del certificado SSL del servidor API de Kubernetes

Configuremos la supervisión de la caducidad de los certificados SSL del servidor API de Kubernetes. Él enviará notificaciones una vez por semana.

Agregue el módulo exportador Blackbox para la autenticación del servidor API de Kubernetes.

kubectl --namespace=monitoring edit configmap prometheus-blackbox-exporter
...
      kube-api:
        http:
          method: GET
          no_follow_redirects: false
          preferred_ip_protocol: ip4
          tls_config:
            insecure_skip_verify: false
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          valid_http_versions:
          - HTTP/1.1
          - HTTP/2
          valid_status_codes: []
        prober: http
        timeout: 5s

Agregar la configuración de raspado de Prometheus

- job_name: 'kube-api-blackbox'
  metrics_path: /probe
  params:
    module: [kube-api]
  static_configs:
   - targets:
      - https://kubernetes.default.svc/api
  relabel_configs:
   - source_labels: [__address__]
     target_label: __param_target
   - source_labels: [__param_target]
     target_label: instance
   - target_label: __address__
     replacement: prometheus-blackbox-exporter:9115 # The blackbox exporter.

Aplicar Prometheus Secret

PROMETHEUS_ADD_CONFIG=$(cat prometheus-additional.yaml | base64)
cat << EOF | kubectl --namespace=monitoring apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: additional-scrape-configs
type: Opaque
data:
  prometheus-additional.yaml: $PROMETHEUS_ADD_CONFIG
EOF

Agregar reglas de alerta

kubectl --namespace=monitoring edit prometheusrules prometheus-k8s-rules
...
  - name: k8s-api-server-cert-expiry
    rules:
    - alert: K8sAPIServerSSLCertExpiringAfterThreeMonths
      expr: probe_ssl_earliest_cert_expiry{job="kube-api-blackbox"} - time() < 86400 * 90 
      for: 1w
      labels:
        severity: warning
      annotations:
        summary: "Kubernetes API Server SSL certificate will expire after three months (instance {{ $labels.instance }})"
        description: "Kubernetes API Server SSL certificate expires in 90 days\n  VALUE = {{ $value }}\n  LABELS: {{ $labels }}"

Enlaces útiles

Monitoreo e inicio de sesión en Docker

Prometheus: monitoreo HTTP a través del exportador Blackbox