Kubernetes: Observability/Health Check

What this article contains :
- What is the need of Observability OR Health check
- How to do Heath check
- What are Probes in term of Kubernetes
- Type of Probes
- Purpose of these probes and how they are different from each other
- Different implementation mechanisms
- How to Configure these different implementations
- Why and how to write named port
- Commands to see these probe status and How to troubleshoot if some problem exit in your config file.
- Few tips to understand 😊
- Summary

Why Heath check is needed?

We all know that Kubernetes can autoscale (up or down)the service instances as per need but how Kubernetes know whether the container inside a pod is up and running?
- Then here comes the term Health check which means continuously keeping track of.
Now the question is what actually is being done here and how?
- Probes

A Probe is a diagnostic performed periodically by the kubelet on a Container. To perform a diagnostic, the kubelet calls a Handler implemented by the container (Implementation mechanism).

Pods are the smallest deployable unit that may contain one or more containers in it. Container probes are small processes that run periodically. The result of these probes(Success, Failed, or Unknown ) determines the container’s state.

Liveness probes try to keep pods healthy by restarting them again, whereas readiness probes make sure that pods are ready to serve requests by receiving them only when all the containers inside that particular pod are ready.
(check below diagrams for both probes to get a better understanding)

Both the probes(readiness and liveness) have 3 mechanisms to get implemented.

Few important Fields that can be used to precisely control the behaviour of liveness and readiness checks are:

• initialDelaySeconds: The number of seconds after the container has started before liveness or readiness probes are initiated.
  The minimum(Defaults) value is 0.
• periodSeconds: How often (in seconds) to perform the probe.
  Default value: 10 seconds
  The minimum value is 1
• Additional fields in case of HTTP GET are host, scheme, path, httpHeaders, port

Readiness probes:

This probe check “Is pod ready to serve the request?”
It runs on the container during its whole lifecycle because a pod is considered as ready when all the container/containers inside the pod are ready.

When the pod is ready(meaning all containers are ready inside it), then in Status will also change from “pending” to “running” (if no problem exists else some other status will be provided)

If a pod’s readiness probe fails, the pod is removed from the Endpoint object meaning now the client connected to the service will not be redirected to this pod.

Adding readiness probe to a pod in your .yaml:

1. Exec:

Containers status is determined by the process’s exit status code.

readinessProbe:
  exec:
    command:
    - cat
    - /tmp/healthy
  initialDelaySeconds: 5
  periodSeconds: 5

2. HTTP GET

It sends the request to the container and the HTTP status code of response determines whether the container is ready or not.

readinessProbe:
  httpGet:
    path: /health
    port: http
  initialDelaySeconds: 5
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 10

TCP Socket

It opens a TCP connection to the specified port of the container. If the connection is established then the container is considered ready.

readinessProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 10

Liveness Probes:

Many applications running for long periods of time eventually transition to broken states, and cannot recover except by being restarted. So, Kubernetes will periodically execute the probe and restart the container.

1. Exec:

It executes a command inside the container and checks the process’s exit status code.
If the status code is o, it is successful but all other codes are considered as failure.

livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy
      initialDelaySeconds: 5
      periodSeconds: 5

2. HTTP GET

It sends the request to the container based on the IP address, port, the path you provide. If the HTTP status code of the response is in between 2xx or 3xx, the probe is considered as successful but if the server returns an error or even if don’t respond it is a failure and time for probe to restart the container.

livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
        httpHeaders:
        - name: Custom-Header
          value: Awesome
      initialDelaySeconds: 3
      periodSeconds: 3

TCP Socket

It opens a TCP connection to the specified port of the container. If the connection is established then the is probe is ok else the container needs to be restarted.

livenessProbe:
      tcpSocket:
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 20

Named Port:

As the topic(Named port) is self-explanatory that we can provide a name to a port and that is called as “Named port”.
But the question is why we need it?
-In the configuration .yaml file, we use the port number at a number of places and its always a good idea to make your file look clean and easy to read along with minimising the error changes which can happen while providing the port number. e.g.

ports:
- name: liveness-port
  containerPort: 8080
  hostPort: 8080

livenessProbe:
  httpGet:
    path: /healthz
    port: liveness-port

Commands to check the status of your pod and troubleshoot if any problem exists in your config:

1. check all pods

$ kubectl get all
 OR 
$ kubectl get pods -n <your NameSpace>

2. If Pod “xyz” fails then Check the detailed information of failed pod

$ kubectl describe pod xyz

3. Check the root cause, what caused this failure.

$ kubectl logs xyz

4. If You want more into it, then shell into the container to get the understanding of what files and created and where they are placed.

5. If you want to write or edit the config file, use

$ vim <.yaml file>

[Note: you can also use Kubernetes dashboard to check pod status]

Tips:
1. Readiness probes are configured similarly to liveness probes. The only difference is that you use the readinessProbe field instead of the livenessProbe field.
2. Use namespace corresponding to pods in command line

Summary :

Readiness and Liveness probes are used for Kubernetes Health check.
Readiness checks whether the container/containers inside the pod are ready to receive Incoming traffic or not while Liveness probe checks whether the container is still alive or not if not then restart it.

[NOTE : startupProbe Indicates whether the application within the container is started. All other probes are disabled if a startup probe is provided until it succeeds. A container does not provide a startup probe, the default state is Success. And that’s the reason we haven’t discussed startupProbe in this article]