How to Monitor Spring Boot Application with Prometheus and Grafana on Kubernetes Cluster

by
with no comment
CloudDevOpsDockerKubernetesmonitoringprometheus

Prometheus and Grafana make it extremely easy to monitor just about any metric in your Kubernetes cluster.

In this Blog I will show How to Monitor Spring Boot Application with Prometheus and Grafana on Kubernetes Cluster

You need a few things.

  1. An existing Kubernetes Cluster with Spring boot application.
  2. kubectl binary locally installed.
  3. Helm Application to manage your Kubernetes Cluster.

Step 1 : Deploy a Spring Boot application on Kubernetes and expose actuator endpoints

In Kubernetes environment , we can configure annotations which will be used by prometheus to scrap data.Below is the complete deployment.yaml file

In which we use image of simple spring boot application ” brainupgrade-in/spring-boot-prometheus:0.0.1-SNAPSHOT “

spring-boot-prometheus-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-boot-prometheus
spec:
  selector:
    matchLabels:
      app: spring-boot-prometheus
  replicas: 1
  template:
    metadata:
      labels:
        app: spring-boot-prometheus
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/path: "/actuator/prometheus"
    spec:
      containers:
        - name: spring-boot-prometheus
          image: brainupgrade.in/spring-boot-prometheus:0.0.1-SNAPSHOT
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
          resources:
            limits:
              memory: 294Mi

Step 2 :Install Helm,The package manager for Kubernetes

Helm makes it extremely easy to make sure you use up to date versions of Prometheus & Grafana & also makes it a-lot easier to deploy and delete.

Helm now has an installer script that will automatically grab the latest version of Helm and install it locally.

You can fetch that script, and then execute it locally. It’s well documented so that you can read through it and understand what it is doing before you run it.

Install helm From Script
$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
$ chmod 700 get_helm.sh
$ ./get_helm.sh

Step 3 :Create separate namespace for Monitoring

Time to monitor

it’s always good idea to keep related things together, We will create separate namespace in Kubernetes for monitoring and will deploy all monitoring related application under that namespace.

Create monitoring namespace with following command

$ kubectl create namespace monitoring

Step 4: Deploy Prometheus using Helm Chart

With the help of Helm, We can deploy prometheus using single command.

$ helm install stable/prometheus --namespace monitoring

This will deploy Prometheus into your cluster in the monitoring namespace and mark the release with the name prometheus.

let’s check if prometheus is running or not using following command :

$ kubectl get pods -n monitoring

Step 5: Deploy Grafana using Helm Chart

When deploying grafana, we need to configure it to read metrics from the right data sources.

There are two ways of achieving this.

  1. Deploy Grafana & add the data source afterwards through the UI.
  2. Add the data source as yaml configs & deploy Grafana. Grafana will use these to automatically configure the data sources when it is provisioned. We are going to take this path as we want everything to be replicate-able without too much manual intervention.

Grafana takes data sources through yaml configs when it starts up. For more information see here: http://docs.grafana.org/administration/provisioning/#datasources

When the Grafana Helm chart gets deployed, it will search for any config maps that contain a grafana_datasource label. So we will add one in our config. That is as follow

config.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-grafana-datasource
  namespace: monitoring
  labels:
    grafana_datasource: '1'
data:
  datasource.yaml: |-
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      access: proxy
      orgId: 1
      url: http://prometheus-server.monitoring.svc.cluster.local

Override Grafana value

When Grafana gets deployed and the provisioner runs, the data source provisioner is deactivated. We need to activate it so it searches for our config maps.

We need to create our own values.yml file to override the datasources search value, so when Grafana is deployed it will search our datasource.yml definition and inject it.

Create a file called values.yml with the following contents:

values.yml
sidecar:
  image: xuxinkun/k8s-sidecar:0.0.7
  imagePullPolicy: IfNotPresent
  datasources:
    enabled: true
    label: grafana_datasource
Apply Config map & Grafana deployment

$ kubectl apply -f config.yml

$ helm install stable/grafana -f values.yml –namespace monitoring –name grafana

( If your having any issue with this command remove value.yaml and simply install grafana we will add dashboard manually in grafana.)

$ helm install stable/grafana –namespace monitoring

And Check that it is running using command :

$ kubectl get pods -n monitoring

Step 6 : Get the Grafana Password

Grafana is deployed with a password. This is good news. But whats the password ?

To know the password run the following command :

$ kubectl get secret –namespace monitoring grafana -o jsonpath=”{.data.admin-password}” | base64 –decode ; echo

Now let’s do port-forward for accessing grafana

$ export POD_NAME=$(kubectl get pods –namespace monitoring -l “app=grafana,release=grafana” -o jsonpath=”{.items[0].metadata.name}”)

$ kubectl –namespace monitoring port-forward $POD_NAME 3000

Go to http://localhost:3000 in your browser. You should see the Grafana login screen:

Login with the username (admin) and password you have from the previous command.

Add a dashboard

Grafana has a long list of prebuilt dashboard here: 
https://grafana.com/dashboards

Here you will find many many dashboards to use. We will use this one as it is quite comprehensive in everything it tracks.

In the left hand menu, choose Dashboards > Manage > + Import

In the Grafana.com dashboard input, add the dashboard ID we want to use: 4701 and click Load

Now as i mention in step 5 if the value.yml is added with deployment you can see datasource here

Here we need to add Proetheus datasource manually

In the left hand menu, Choose Configuration > Data Source > Add Data Source 

And Select Time Series Database Prometheus

Configure Prometheus server URL with the kubernetes service endpoint

In HTTP URL Block insert this URL : http://prometheus-server:80

Now Click on Save and Test button at bottom :

The list of metrics is extensive. Go over them and see what is useful, copy their structures and panels and create your own dashboards for the big screens in the office !

Enjoy !

Related posts:

  1. Microservices Observability on Kubernetes
  2. Digital Transformation – a Journey with Automation, Tools, Process, Cloud, Containers Technologies.
  3. Scalable Jenkins on Kubernetes Cluster & Pipeline for Automated CI CD (Jenkins – Github Integration)
  4. Expose your app to the Internet using Ingress Controller & save 1000s of dollars on AWS Load Balancer
Tags: No tags

Add a Comment

Your email address will not be published. Required fields are marked *