Monitoring is a critical aspect of managing and maintaining a Kubernetes cluster. Prometheus is a powerful open-source monitoring and alerting toolkit designed specifically for reliability and scalability. In this section, we will cover the basics of Prometheus, how to set it up in a Kubernetes cluster, and how to use it to monitor your applications and infrastructure.

Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. It is now a standalone open-source project and maintained independently of any company. Prometheus is designed for reliability and scalability, making it a popular choice for monitoring Kubernetes clusters.

• Multi-dimensional data model: Time series data is identified by metric name and key/value pairs.
• Flexible query language: PromQL (Prometheus Query Language) allows for powerful querying and aggregation.
• Pull-based model: Prometheus scrapes metrics from instrumented jobs.
• No reliance on distributed storage: Single server nodes are autonomous.
• Built-in support for service discovery: Integrates with Kubernetes, Consul, and other service discovery mechanisms.
• Alerting: Integrated alerting based on PromQL queries.

To set up Prometheus in a Kubernetes cluster, you can use the Prometheus Operator, which simplifies the deployment and management of Prometheus instances. The Prometheus Operator provides Kubernetes native deployment and management of Prometheus and related monitoring components.

1. Install Prometheus Operator:

   • You can install the Prometheus Operator using Helm, a package manager for Kubernetes.

   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo update
   helm install prometheus prometheus-community/kube-prometheus-stack
   

2. Verify Installation:

   • Check the status of the Prometheus pods to ensure they are running.

   kubectl get pods -n default -l "release=prometheus"
   

3. Access Prometheus Dashboard:

   • Forward the Prometheus service port to your local machine to access the Prometheus dashboard.

   kubectl port-forward svc/prometheus-kube-prometheus-prometheus 9090:9090
   

   • Open your browser and navigate to http://localhost:9090 to access the Prometheus dashboard.

Once Prometheus is set up, you can start monitoring your Kubernetes cluster and applications. Prometheus collects metrics from various sources, including Kubernetes components, application pods, and custom metrics.

• Node Metrics: CPU usage, memory usage, disk I/O, network I/O.
• Pod Metrics: CPU usage, memory usage, restart counts, pod status.
• Cluster Metrics: API server request rates, scheduler performance, controller manager metrics.

1. CPU Usage of a Node:

   sum(rate(node_cpu_seconds_total{mode!="idle"}[5m])) by (instance)
   

2. Memory Usage of a Pod:

   sum(container_memory_usage_bytes{pod="your-pod-name"}) by (container)
   

3. Total Requests to the API Server:

   sum(rate(apiserver_request_total[5m]))
   

Prometheus allows you to create alerts based on specific conditions. Alerts are defined in a configuration file and evaluated at regular intervals.

Example Alert Rule:

Create a file named alert-rules.yaml with the following content:

Apply the alert rule:

1. Deploy a Sample Application:

   • Deploy a sample Nginx application in your Kubernetes cluster.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: nginx-deployment
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: nginx
     template:
       metadata:
         labels:
           app: nginx
       spec:
         containers:
         - name: nginx
           image: nginx:1.14.2
           ports:
           - containerPort: 80
   

   kubectl apply -f nginx-deployment.yaml
   

2. Expose the Application:

   • Create a service to expose the Nginx application.

   apiVersion: v1
   kind: Service
   metadata:
     name: nginx-service
   spec:
     selector:
       app: nginx
     ports:
       - protocol: TCP
         port: 80
         targetPort: 80
   

   kubectl apply -f nginx-service.yaml
   

3. Monitor the Application:

   • Use Prometheus to monitor the CPU and memory usage of the Nginx pods.

   sum(rate(container_cpu_usage_seconds_total{pod=~"nginx-deployment-.*"}[1m])) by (pod)
   

   sum(container_memory_usage_bytes{pod=~"nginx-deployment-.*"}) by (pod)
   

• Deploy the Nginx application and expose it using the provided YAML files.
• Access the Prometheus dashboard and use the provided PromQL queries to monitor the CPU and memory usage of the Nginx pods.

In this section, we covered the basics of Prometheus, how to set it up in a Kubernetes cluster, and how to use it to monitor your applications and infrastructure. We also explored some key metrics to monitor and how to create alerts based on specific conditions. By the end of this section, you should have a good understanding of how to use Prometheus for monitoring in Kubernetes.