Kubernetes Monitoring and Observability: A Complete Guide

Effective monitoring and observability are crucial for maintaining healthy Kubernetes clusters. This guide covers essential monitoring strategies, tools, and best practices for gaining deep insights into your Kubernetes infrastructure and applications.

Monitoring Architecture

A comprehensive Kubernetes monitoring setup typically involves multiple components working together to collect, store, and visualize metrics, logs, and traces.

Prometheus Setup

Prometheus is the de facto standard for Kubernetes monitoring. Here's how to set it up using Helm:

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
    scrape_configs:
      - job_name: 'kubernetes-nodes'
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          - source_labels: [__meta_kubernetes_node_name]
            target_label: node

Key Metrics to Monitor

Grafana Dashboards

Grafana provides powerful visualization capabilities for your monitoring data. Here's an example dashboard configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboard
data:
  kubernetes-cluster.json: |
    {
      "dashboard": {
        "title": "Kubernetes Cluster Overview",
        "panels": [
          {
            "title": "CPU Usage",
            "type": "graph",
            "datasource": "Prometheus",
            "targets": [
              {
                "expr": "sum(rate(container_cpu_usage_seconds_total{container!=\"\"}[5m])) by (pod)"
              }
            ]
          }
        ]
      }
    }

Log Aggregation

Centralized logging is essential for troubleshooting and monitoring. Here's how to set up logging with Loki:

Promtail Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
data:
  promtail.yaml: |
    server:
      http_listen_port: 9080
    positions:
      filename: /run/promtail/positions.yaml
    clients:
      - url: http://loki:3100/loki/api/v1/push
    scrape_configs:
      - job_name: kubernetes-pods
        kubernetes_sd_configs:
          - role: pod

Alerting Configuration

Set up effective alerting to proactively respond to issues:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kubernetes-alerts
spec:
  groups:
    - name: kubernetes
      rules:
        - alert: HighCPUUsage
          expr: sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (pod) > 0.8
          for: 5m
          labels:
            severity: warning
          annotations:
            description: "Pod {{ $labels.pod }} has high CPU usage"

Distributed Tracing

Implement distributed tracing with OpenTelemetry and Jaeger:

OpenTelemetry Configuration

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: cluster-collector
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
    processors:
      batch:
    exporters:
      jaeger:
        endpoint: jaeger-collector:14250
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [jaeger]

Best Practices

1. Resource Monitoring

   • Set appropriate thresholds based on historical usage patterns
   • Monitor both cluster and application metrics to get a complete picture
   • Implement predictive scaling using metrics-based automation
   • Configure resource quotas and limits for namespaces
   • Set up alerts for resource saturation

2. Log Management

   • Use structured logging with consistent formats
   • Implement log rotation to manage storage efficiently
   • Set retention policies based on compliance requirements
   • Configure log aggregation with proper indexing
   • Implement log level filtering for different environments

3. Alert Configuration

   • Define clear severity levels (P0, P1, P2, etc.)
   • Set up proper alert routing and escalation policies
   • Implement alert grouping to prevent alert fatigue
   • Configure alert deduplication
   • Document response procedures for each alert type

4. Performance Monitoring

   • Monitor latency across service boundaries
   • Track error rates and success ratios
   • Monitor throughput and request rates
   • Set up SLO/SLI monitoring
   • Implement performance baselines

5. Security Monitoring

   • Monitor authentication and authorization events
   • Track configuration changes
   • Implement audit logging
   • Monitor network policies
   • Set up vulnerability scanning alerts