This content originally appeared on DEV Community and was authored by Subham Nandi
1. Prerequisites
Please ensure you have the following prerequisites:
- A Kubernetes cluster running (EKS, GKE, Minikube, etc.)
-
kubectlCLI installed and configured to communicate with the Kubernetes cluster - Helm installed on your machine for deploying the Prometheus stack
2. Install Prometheus Stack Using Helm
We will use Helm to install the Prometheus stack, which includes Prometheus, Grafana, Alertmanager, and other components like kube-state-metrics and node_exporter.
- Add the Helm Repository:
First, add the prometheus-community Helm chart repository.
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
- Install the Prometheus Stack:
Now, install the Prometheus stack. This command will install Prometheus, Grafana, Alertmanager, and other related services in a monitoring namespace.
kubectl create namespace monitoring
helm install prometheus-stack prometheus-community/kube-prometheus-stack --namespace monitoring
This installs all components, including:
- Prometheus for scraping and storing metrics
- Grafana for visualizing metrics
- Alertmanager for managing alerts
- kube-state-metrics for Kubernetes metrics
- node_exporter for node-level metrics
- Verify the Installation:
Run the following command to ensure all pods are up and running:
kubectl get pods -n monitoring
You should see pods for Prometheus, Grafana, Alertmanager, node_exporter, and kube-state-metrics.
3. Expose Services Using kubectl port-forward
Since your Kubernetes cluster might not have a LoadBalancer or Ingress configured, we will use kubectl port-forward to access the Prometheus, Grafana, and Alertmanager web interfaces.
Expose Prometheus:
kubectl port-forward svc/prometheus-stack-kube-prometheus-prometheus -n monitoring 9090:9090
- This command forwards local port
9090to Prometheus running in Kubernetes. - You can access Prometheus by opening
http://localhost:9090in your browser.
Expose Grafana:
kubectl port-forward svc/prometheus-stack-grafana -n monitoring 3000:80
- This command forwards local port
3000to Grafana running in Kubernetes. -
You can access Grafana by opening
http://localhost:3000in your browser.-
Grafana Credentials: The default username is
adminand the password can be obtained by running:
kubectl get secret --namespace monitoring prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo -
Grafana Credentials: The default username is
Expose Alertmanager:
kubectl port-forward svc/prometheus-stack-kube-prometheus-alertmanager -n monitoring 9093:9093
- This command forwards local port
9093to Alertmanager running in Kubernetes. - You can access Alertmanager by opening
http://localhost:9093in your browser.
4. Understanding Exporters and Metrics Collection
Prometheus collects metrics from Kubernetes using exporters. Two critical exporters included in the Prometheus stack are:
- node_exporter: Provides hardware and OS-level metrics, such as CPU, memory, and disk usage for each Kubernetes node.
- kube-state-metrics: Collects Kubernetes object metrics, like pod status, node status, container restarts, etc.
Checking node_exporter and kube-state-metrics Pods:
To verify that these exporters are running:
kubectl get pods -n monitoring -l app.kubernetes.io/name=node-exporter
kubectl get pods -n monitoring -l app.kubernetes.io/name=kube-state-metrics
These should display the respective pods.
Accessing node_exporter Metrics:
node_exporter runs as a DaemonSet (i.e., one pod per node). It exposes metrics about each node in the cluster.
To check metrics for node_exporter:
- First, get the service IP of
node_exporter:
kubectl get svc -n monitoring
Look for prometheus-stack-kube-prometheus-node-exporter, which will have a ClusterIP.
- To access
node_exportermetrics:
kubectl port-forward svc/prometheus-stack-kube-prometheus-node-exporter -n monitoring 9100:9100
- Then, open
http://localhost:9100/metricsto view the raw metrics exposed bynode_exporter.
Accessing kube-state-metrics:
kube-state-metrics provides information about the state of Kubernetes objects (pods, services, deployments, etc.).
To access kube-state-metrics:
- Get the service IP for
kube-state-metrics:
kubectl get svc -n monitoring
Look for prometheus-stack-kube-state-metrics, which will have a ClusterIP.
- To access
kube-state-metricsmetrics:
kubectl port-forward svc/prometheus-stack-kube-state-metrics -n monitoring 8080:8080
- Then, open
http://localhost:8080/metricsto view the raw metrics exposed bykube-state-metrics.
5. Querying Metrics in Prometheus
Prometheus provides a powerful query language called PromQL to extract and analyze metrics. Let's see some examples:
- Get total container restarts in the cluster:
kube_pod_container_status_restarts_total
This query returns how many times containers have restarted in all namespaces.
-
Filter restarts for a specific namespace (e.g.,
default):
kube_pod_container_status_restarts_total{namespace="default"}
- Get CPU usage for all nodes:
node_cpu_seconds_total
6. Create Grafana Dashboards
Grafana helps visualize the metrics collected by Prometheus. Here’s how to configure it:
- Log in to Grafana:
Access Grafana at http://localhost:3000 using the default credentials (admin/admin).
- Add Prometheus as a Data Source:
- Navigate to Configuration > Data Sources.
- Click Add Data Source.
- Select Prometheus.
- Enter
http://prometheus-stack-kube-prometheus-prometheus.monitoring.svc:9090as the URL. - Click Save & Test.
- Import Pre-built Dashboards:
Grafana provides pre-built dashboards for Kubernetes. You can find and import them from the Grafana website or within Grafana itself.
This content originally appeared on DEV Community and was authored by Subham Nandi
Subham Nandi | Sciencx (2024-10-22T11:39:15+00:00) Day 3 – Prometheus and Grafana explained. Retrieved from https://www.scien.cx/2024/10/22/day-3-prometheus-and-grafana-explained/
Please log in to upload a file.
There are no updates yet.
Click the Upload button above to add an update.