The Easiest Way to Debug Kubernetes Workloads

This article introduces debugging methods such as Kubectl debug and ephemeral containers.

Author: Martin Heinz Translation: Bach (K8sMeetup)

Proofread: Xingkongxia de Wenzi Source: K8sMeetup Community

Debugging containerized workloads and Pods is a routine task for every developer and DevOps engineer using Kubernetes. Typically, we simply use kubectl logs or kubectl describe pod to find problems, but sometimes, certain issues can be particularly difficult to troubleshoot. In such cases, one might try using kubectl exec, but sometimes that won’t work either, as containers like Distroless do not even allow SSH access to a shell. So, what do we do if all the above methods fail?

A Better Method

Actually, we just need to use a more appropriate tool. If debugging workloads on Kubernetes, the appropriate tool is kubectl debug. This is a new command added recently (v1.18) that allows debugging running pods. It injects a special container called EphemeralContainer into the problematic Pod, allowing us to inspect and troubleshoot.kubectl debug looks great, but it requires ephemeral containers; what exactly are ephemeral containers?

Ephemeral containers are actually a subresource of a Pod, similar to a regular container. However, unlike regular containers, ephemeral containers are not used to build applications but rather for inspection. We do not define them when creating the Pod, but instead use a special API to inject them into the running Pod to execute commands and inspect the Pod environment. Apart from these differences, ephemeral containers also lack some fields that regular containers have, such as ports and resources.

So why not just use a regular container? This is because we cannot add regular containers to a Pod; they should be one-off (need to be deleted or recreated at any time), which makes reproducing the errors of the problematic Pod difficult and troubleshooting cumbersome. This is the reason ephemeral containers were added to the API—they allow us to add ephemeral containers to existing Pods, thus enabling inspection of running Pods.

Although ephemeral containers are part of the core Pod specification in Kubernetes, many people may not have heard of them. This is because ephemeral containers are in early Alpha stages, meaning they are not enabled by default. Resources and features in the Alpha stage may undergo significant changes or be completely removed in a future version of Kubernetes. Therefore, to use them, they must be explicitly enabled in kubelet using Feature Gate.

Configuring Feature Gates

Now, if you are determined to try kubectl debug, how do you enable the feature gate for ephemeral containers? It depends on the cluster setup. For example, if you are creating a cluster using kubeadm now, you can use the following cluster configuration to enable ephemeral containers:

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.20.2
apiServer:
  extraArgs:
    feature-gates: EphemeralContainers=true

In the following example, for simplicity and testing purposes, we use a KinD (Kubernetes in Docker) cluster, which allows us to specify the feature gates to enable. Create our test cluster:

# File: config.yaml
# Run:  kind create cluster --config ./config.yaml --name kind --image=kindest/node:v1.20.2
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  EphemeralContainers: true
nodes:
- role: control-plane

With the cluster running, we need to verify its validity. The simplest way is to check the Pod API, which should now include the ephemeral containers section along with the usual containers:

~ $ kubectl explain pod.spec.ephemeralContainers
KIND:     Pod
VERSION:  v1

RESOURCE: ephemeralContainers &lt;[]Object&gt;

DESCRIPTION:
     List of ephemeral containers run in this pod....
...

Now that everything is set up, we can start using kubectl debug. Let’s start with a simple example:

~ $ kubectl run some-app --image=k8s.gcr.io/pause:3.1 --restart=Never
~ $ kubectl debug -it some-app --image=busybox --target=some-app
Defaulting debug container name to debugger-tfqvh.
If you don't see a command prompt, try pressing enter.
/ #
# From other terminal...
~ $ kubectl describe pod some-app
...
Containers:
  some-app:
    Container ID:   containerd://60cc537eee843cb38a1ba295baaa172db8344eea59de4d75311400436d4a5083
    Image:          k8s.gcr.io/pause:3.1
    Image ID:       k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea
...
Ephemeral Containers:
  debugger-tfqvh:
    Container ID:   containerd://12efbbf2e46bb523ae0546b2369801b51a61e1367dda839ce0e02f0e5c1a49d6
    Image:          busybox
    Image ID:       docker.io/library/busybox@sha256:ce2360d5189a033012fbad1635e037be86f23b65cfd676b436d0931af390a2ac
    Port:           &lt;none&gt;
    Host Port:      &lt;none&gt;
    State:          Running
      Started:      Mon, 15 Mar 2021 20:33:51 +0100
    Ready:          False
    Restart Count:  0
    Environment:    &lt;none&gt;
    Mounts:         &lt;none&gt;

We first start a Pod named some-app for debugging. Then we run kubectl debug against this Pod, specifying busybox as the image for the ephemeral container and targeting the original container. Additionally, we need to include the -it flag so that we can immediately attach to the container and get a shell session.

In the code above, you can see that if we run kubectl debug on the Pod and then describe it, its description will include the ephemeral container section that was previously specified as a command option value.

Process Namespace Sharing

kubectl debug is a very powerful tool, but sometimes adding a container to the Pod is not enough to get application-related information running in another container of the Pod. This may be the case when the faulty container does not include the necessary debugging tools or even a shell. In this case, we can use Process Sharing to inspect the original container of the Pod using the injected ephemeral container.

One issue with process sharing is that it cannot be applied to existing Pods, so we must create a new Pod with its spec.shareProcessNamespace set to true, and inject an ephemeral container into it. This can be a bit cumbersome, especially when needing to debug multiple Pods or containers, or when needing to repeat the operation. Fortunately, kubectl debug can do this using --share-processes:

~ $ kubectl run some-app --image=nginx --restart=Never
~ $ kubectl debug -it some-app --image=busybox --share-processes --copy-to=some-app-debug
Defaulting debug container name to debugger-tkwst.
If you don't see a command prompt, try pressing enter.
/ # ps ax
PID   USER     TIME  COMMAND
    1 root      0:00 /pause
    8 root      0:00 nginx: master process nginx -g daemon off;
   38 101       0:00 nginx: worker process
   39 root      0:00 sh
   46 root      0:00 ps ax
~ $ cat /proc/8/root/etc/nginx/conf.d/default.conf 
server {
    listen       80;
    listen  [::]:80;
    server_name  localhost;
...

The code above shows that through process sharing, we can see everything inside another container in the Pod, including its processes and files, which is very convenient for debugging. Additionally, apart from --share-processes, there is also --copy-to=new-pod-name, as we need to create a new Pod whose name is specified by this flag. If we list the running Pods from another terminal, we will see the following:

# From other terminal:
~ $ kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
some-app         1/1     Running   0          23h
some-app-debug   2/2     Running   0          20s

This is our new debugging Pod on the original application Pod. Compared to the original container, it has 2 containers because it also includes the ephemeral container. Moreover, if we want to verify at any time whether process sharing is allowed in the Pod, we can run:

~ $ kubectl get pod some-app-debug -o json  | jq .spec.shareProcessNamespace
true

Use it Wisely

Now that we have enabled the feature and know how the command works, let’s try using it and debugging some applications. Imagine a scenario where we have a problematic application, and we need to troubleshoot network-related issues in its container. The application does not have the necessary network CLI tools available for us to use. To solve this problem, we use kubectl debug as follows:

~ $ kubectl run distroless-python --image=martinheinz/distroless-python --restart=Never
~ $ kubectl exec -it distroless-python -- /bin/sh
# id
/bin/sh: 1: id: not found
# ls
/bin/sh: 2: ls: not found
# env
/bin/sh: 3: env: not found
#
...

kubectl debug -it distroless-python --image=praqma/network-multitool --target=distroless-python -- sh
Defaulting debug container name to debugger-rvtd4.
If you don't see a command prompt, try pressing enter.
/ # ping localhost
PING localhost(localhost (::1)) 56 data bytes
64 bytes from localhost (::1): icmp_seq=1 ttl=64 time=0.025 ms
64 bytes from localhost (::1): icmp_seq=2 ttl=64 time=0.044 ms
64 bytes from localhost (::1): icmp_seq=3 ttl=64 time=0.027 ms

After starting a Pod, we first try to get a shell session into its container, which seems to work, but when we try to run some basic commands, we find that nothing is available there. Therefore, we use praqma/network-multitool to inject an ephemeral container into the Pod, which contains tools like curl, ping, telnet, and now we can perform all necessary troubleshooting.

In the example above, it was sufficient for us to enter another container in the Pod. However, sometimes we may need to look directly at the problematic container. In such cases, we can use process sharing like this:

~ $ kubectl run distroless-python --image=martinheinz/distroless-python --restart=Never
~ $ kubectl debug -it distroless-python --image=busybox --share-processes --copy-to=distroless-python-debug
Defaulting debug container name to debugger-l692h.
If you don't see a command prompt, try pressing enter.
/ # ps ax
PID   USER     TIME  COMMAND
    1 root      0:00 /pause
    8 root      0:00 /usr/bin/python3.5 sleep.py  # Original container is just sleeping forever
   14 root      0:00 sh
   20 root      0:00 ps ax
/ # cat /proc/8/root/app/sleep.py 
import time
print("sleeping for 1 hour")
time.sleep(3600)

Here, we again run a container using the Distroless image. We cannot do anything in its shell. We run kubectl debug along with --share-processes --copy-to=..., which creates a new Pod with the additional ephemeral container that has access to all processes. When we list the running processes, we can see the application container’s process with PID 8, which we can use to explore files and environment. To do this, we need to go through the /proc/<PID>/... directory, which in this example is /proc/8/root/app/....

Another common scenario is when an application continuously crashes upon container startup, making it very difficult to debug since there is not enough time to import a shell session into the container and run troubleshooting commands. In this case, the solution is to create a container with a different entry point and command that prevents the application from crashing immediately and allows us to debug:

~ $ kubectl get pods
NAME                READY   STATUS             RESTARTS   AGE
crashing-app        0/1     CrashLoopBackOff   1          8s

~ $ kubectl debug crashing-app -it --copy-to=crashing-app-debug --container=crashing-app -- sh
If you don't see a command prompt, try pressing enter.
# id
uid=0(root) gid=0(root) groups=0(root)
#
...
# From another terminal
~ $ kubectl get pods
NAME                READY   STATUS             RESTARTS   AGE
crashing-app        0/1     CrashLoopBackOff   3          2m7s
crashing-app-debug  1/1     Running            0          16s

Debugging Cluster Nodes

This article mainly focuses on debugging Pods and their containers, but any cluster administrator knows that often what needs debugging is the nodes rather than the Pods. Fortunately, kubectl debug allows debugging nodes by creating a Pod that runs on the specified node, with the node’s root filesystem mounted at /root. We can even access host binaries using chroot, essentially acting as an SSH connection to the node:

~ $ kubectl get nodes
NAME                 STATUS   ROLES                  AGE   VERSION
kind-control-plane   Ready    control-plane,master   25h   v1.20.2

~ $ kubectl debug node/kind-control-plane -it --image=ubuntu
Creating debugging pod node-debugger-kind-control-plane-hvljt with container debugger on node kind-control-plane.
If you don't see a command prompt, try pressing enter.
root@kind-control-plane:/# chroot /host
# head kind/kubeadm.conf
apiServer:
  certSANs:
  - localhost
  - 127.0.0.1
  extraArgs:
    feature-gates: EphemeralContainers=true
    runtime-config: ""
apiVersion: kubeadm.k8s.io/v1beta2
clusterName: kind
controlPlaneEndpoint: kind-control-plane:6443

In the code above, we first identify the node we want to debug, and then explicitly run kubectl debug using node/... as an argument to access the node of our cluster. After that, when connected to the Pod, we use chroot /host to break through the chroot and fully enter the host. Finally, to verify that we can indeed see everything on the host, we check part of the kubeadm.conf and ultimately see the configuration we set at the beginning of the article: feature-gates: EphemeralContainers=true.

Conclusion

Being able to debug applications and services quickly and effectively can save a lot of time, but more importantly, it can greatly assist in resolving issues that, if not addressed immediately, could end up costing a lot of money. This is why tools like kubectl debug are essential to have at our disposal, even if they are not formally released or enabled by default. If enabling ephemeral containers is not an option, trying alternative debugging methods may be a good idea, such as using a debug version of an application image that includes troubleshooting tools; or temporarily changing the container command of the Pod to prevent it from crashing.

Original link:https://towardsdatascience.com/the-easiest-way-to-debug-kubernetes-workloads-ff2ff5e3cc75

END

The Easiest Way to Debug Kubernetes Workloads

Related posts:

Leave a Comment Cancel reply