The Easiest Way to Debug Kubernetes Workloads

This article introduces debugging methods such as Kubectl debug and ephemeral containers.

Author: Martin Heinz Translation:Bach (K8sMeetup)

Proofread by: Xingkong under the stars Source: K8sMeetup community

Debugging containerized workloads and Pods is a daily task for every developer and DevOps engineer using Kubernetes. Typically, we simply use kubectl logs or kubectl describe pod to find out what went wrong, but sometimes certain issues can be particularly hard to trace. In such cases, people may try using kubectl exec, but sometimes that doesn’t work either, as containers like Distroless do not even allow SSH access to a shell. So, what should we do if all the above methods fail?

A Better Method

Actually, we just need to use a more appropriate tool. If debugging workloads on Kubernetes, the appropriate tool is kubectl debug. This is a new command added recently (v1.18) that allows debugging of running pods. It injects a special container called EphemeralContainer into the problematic Pod, allowing us to inspect and troubleshoot.kubectl debug looks great, but it requires ephemeral containers; what exactly are ephemeral containers?

Ephemeral containers are actually a sub-resource of a Pod, similar to a normal container. However, unlike normal containers, ephemeral containers are not used to build applications but rather for inspection. We do not define them when creating a Pod; instead, we inject them into the running Pod using a special API to run commands and inspect the Pod environment. In addition to these differences, ephemeral containers also lack some fields that normal containers have, such as ports and resources.

So why can’t we just use normal containers? This is because we cannot add normal containers to a Pod; they should be one-time (need to be deleted or recreated at any time), which makes it difficult to reproduce the errors of the problematic Pod, making troubleshooting cumbersome. This is the reason for adding ephemeral containers to the API—they allow us to add ephemeral containers to existing Pods for inspecting running Pods.

Although ephemeral containers are part of the core Pod specification of Kubernetes, many people may not have heard of them. This is because ephemeral containers are in an early Alpha stage, meaning they are not enabled by default. Alpha-stage resources and features may undergo significant changes or may be completely removed in a future version of Kubernetes. Therefore, to use them, they must be explicitly enabled in the kubelet using Feature Gates.

Configuring Feature Gates

Now, if we are sure we want to try kubectl debug, how do we enable the feature gate for ephemeral containers? This depends on the cluster setup. For example, if we are currently using kubeadm to create the cluster, we can use the following cluster configuration to enable ephemeral containers:

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.20.2
apiServer:
  extraArgs:
    feature-gates: EphemeralContainers=true

In the following example, for simplicity and testing purposes, we use a KinD (Kubernetes in Docker) cluster, which allows us to specify the feature gates to enable. Creating our test cluster:

# File: config.yaml
# Run:  kind create cluster --config ./config.yaml --name kind --image=kindest/node:v1.20.2
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  EphemeralContainers: true
nodes:
- role: control-plane

With the cluster running, we need to verify its validity. The simplest way is to check the Pod API, which should now contain the ephemeral containers section along with the usual containers:

~ $ kubectl explain pod.spec.ephemeralContainers
KIND:     Pod
VERSION:  v1

RESOURCE: ephemeralContainers &lt;[]Object&gt;

DESCRIPTION:
     List of ephemeral containers run in this pod....
...

Now that everything is set up, we can start using kubectl debug. Let’s start with a simple example:

~ $ kubectl run some-app --image=k8s.gcr.io/pause:3.1 --restart=Never
~ $ kubectl debug -it some-app --image=busybox --target=some-app
Defaulting debug container name to debugger-tfqvh.
If you don't see a command prompt, try pressing enter.
/ #
# From other terminal...
~ $ kubectl describe pod some-app
...
Containers:
  some-app:
    Container ID:   containerd://60cc537eee843cb38a1ba295baaa172db8344eea59de4d75311400436d4a5083
    Image:          k8s.gcr.io/pause:3.1
    Image ID:       k8s.gcr.io/pause@sha256:f78411e19d84a252e53bff71a4407a5686c46983a2c2eeed83929b888179acea
...
Ephemeral Containers:
  debugger-tfqvh:
    Container ID:   containerd://12efbbf2e46bb523ae0546b2369801b51a61e1367dda839ce0e02f0e5c1a49d6
    Image:          busybox
    Image ID:       docker.io/library/busybox@sha256:ce2360d5189a033012fbad1635e037be86f23b65cfd676b436d0931af390a2ac
    Port:           &lt;none&gt;
    Host Port:      &lt;none&gt;
    State:          Running
      Started:      Mon, 15 Mar 2021 20:33:51 +0100
    Ready:          False
    Restart Count:  0
    Environment:    &lt;none&gt;
    Mounts:         &lt;none&gt;

We first started a Pod named some-app for debugging. We then run kubectl debug against this Pod, specifying busybox as the image for the ephemeral container and targeting the original container. Additionally, we need to include the -it parameter so that we can immediately attach to the container and get a shell session.

In the code above, we can see that after running kubectl debug on the Pod and describing it, its description will include the ephemeral container section with the previously specified command option value.

Process Namespace Sharing

kubectl debug is a very powerful tool, but sometimes adding a container to a Pod is not enough to get application-related information running in another container of the Pod. This may be the case when the faulty container does not include necessary debugging tools or even a shell. In such cases, we can use Process Sharing to inspect the original container of the Pod using the injected ephemeral container.

One issue with process sharing is that it cannot be applied to existing Pods, so we must create a new Pod, set its spec.shareProcessNamespace to true, and inject an ephemeral container into it. This can be a bit tedious, especially when debugging multiple Pods or containers, or when needing to repeat the operation. Fortunately, kubectl debug can do this with --share-processes:

~ $ kubectl run some-app --image=nginx --restart=Never
~ $ kubectl debug -it some-app --image=busybox --share-processes --copy-to=some-app-debug
Defaulting debug container name to debugger-tkwst.
If you don't see a command prompt, try pressing enter.
/ # ps ax
PID   USER     TIME  COMMAND
    1 root      0:00 /pause
    8 root      0:00 nginx: master process nginx -g daemon off;
   38 101       0:00 nginx: worker process
   39 root      0:00 sh
   46 root      0:00 ps ax
~ $ cat /proc/8/root/etc/nginx/conf.d/default.conf 
server {
    listen       80;
    listen  [::]:80;
    server_name  localhost;
...

The code above shows that through process sharing, we can see everything inside another container in the Pod, including its processes and files, which is very convenient for debugging. Additionally, besides --share-processes, we also included --copy-to=new-pod-name, as we need to create a new Pod whose name is specified by that flag. If we list the running Pods from another terminal, we will see the following:

# From other terminal:
~ $ kubectl get pods
NAME             READY   STATUS    RESTARTS   AGE
some-app         1/1     Running   0          23h
some-app-debug   2/2     Running   0          20s

This is our new debugging Pod on the original application Pod. It has 2 containers compared to the original container because it also includes the ephemeral container. Additionally, if we want to verify whether process sharing is allowed in the Pod at any time, we can run:

~ $ kubectl get pod some-app-debug -o json  | jq .spec.shareProcessNamespace
true

Use It Wisely

Now that we have enabled the feature and know how the commands work, let’s try using it and debug some applications. Imagine a scenario where we have a problematic application, and we need to troubleshoot network-related issues in its container. The application does not have the necessary network CLI tools that we can use. To resolve this issue, we use kubectl debug as follows:

~ $ kubectl run distroless-python --image=martinheinz/distroless-python --restart=Never
~ $ kubectl exec -it distroless-python -- /bin/sh
# id
/bin/sh: 1: id: not found
# ls
/bin/sh: 2: ls: not found
# env
/bin/sh: 3: env: not found
#
...

kubectl debug -it distroless-python --image=praqma/network-multitool --target=distroless-python -- sh
Defaulting debug container name to debugger-rvtd4.
If you don't see a command prompt, try pressing enter.
/ # ping localhost
PING localhost(localhost (::1)) 56 data bytes
64 bytes from localhost (::1): icmp_seq=1 ttl=64 time=0.025 ms
64 bytes from localhost (::1): icmp_seq=2 ttl=64 time=0.044 ms
64 bytes from localhost (::1): icmp_seq=3 ttl=64 time=0.027 ms

After starting a Pod, we first try to get a shell session into its container, which seems effective, but when we try to run some basic commands, we find that nothing is available there. So, we use praqma/network-multitool to inject an ephemeral container into the Pod, which includes tools such as curl, ping, telnet, and now we can perform all necessary troubleshooting.

In the example above, it was sufficient for us to enter another container of the Pod. But sometimes, we may need to look directly at the problematic container. In such cases, we can use process sharing like this:

~ $ kubectl run distroless-python --image=martinheinz/distroless-python --restart=Never
~ $ kubectl debug -it distroless-python --image=busybox --share-processes --copy-to=distroless-python-debug
Defaulting debug container name to debugger-l692h.
If you don't see a command prompt, try pressing enter.
/ # ps ax
PID   USER     TIME  COMMAND
    1 root      0:00 /pause
    8 root      0:00 /usr/bin/python3.5 sleep.py  # Original container is just sleeping forever
   14 root      0:00 sh
   20 root      0:00 ps ax
/ # cat /proc/8/root/app/sleep.py 
import time
print("sleeping for 1 hour")
time.sleep(3600)

Here, we again run a container using a Distroless image. We cannot do anything in its shell. We run kubectl debug with --share-processes --copy-to=..., which creates a new Pod with an additional ephemeral container that can access all processes. When we list the running processes, we can see the application container’s process with PID 8, which we can use to explore files and the environment. To do this, we need to go through the /proc/<PID>/... directory, in this case, /proc/8/root/app/....

Another common situation is when an application crashes continuously upon container startup, making debugging very difficult since there is not enough time to import a shell session into the container and run troubleshooting commands. In this case, the solution is to create a container with a different entry point and command that can prevent the application from crashing immediately and allow us to debug:

~ $ kubectl get pods
NAME                READY   STATUS             RESTARTS   AGE
crashing-app        0/1     CrashLoopBackOff   1          8s

~ $ kubectl debug crashing-app -it --copy-to=crashing-app-debug --container=crashing-app -- sh
If you don't see a command prompt, try pressing enter.
# id
uid=0(root) gid=0(root) groups=0(root)
#
...
# From another terminal
~ $ kubectl get pods
NAME                READY   STATUS             RESTARTS   AGE
crashing-app        0/1     CrashLoopBackOff   3          2m7s
crashing-app-debug  1/1     Running            0          16s

Debugging Cluster Nodes

This article mainly focuses on debugging Pods and their containers, but any cluster administrator knows that often what needs debugging is the nodes rather than the Pods. Fortunately, kubectl debug allows debugging nodes by creating a Pod that will run on the specified node, with the node’s root filesystem mounted in the /root directory. We can even access the host binaries using chroot, essentially acting as an SSH connection to the node:

~ $ kubectl get nodes
NAME                 STATUS   ROLES                  AGE   VERSION
kind-control-plane   Ready    control-plane,master   25h   v1.20.2

~ $ kubectl debug node/kind-control-plane -it --image=ubuntu
Creating debugging pod node-debugger-kind-control-plane-hvljt with container debugger on node kind-control-plane.
If you don't see a command prompt, try pressing enter.
root@kind-control-plane:/# chroot /host
# head kind/kubeadm.conf
apiServer:
  certSANs:
  - localhost
  - 127.0.0.1
  extraArgs:
    feature-gates: EphemeralContainers=true
    runtime-config: ""
apiVersion: kubeadm.k8s.io/v1beta2
clusterName: kind
controlPlaneEndpoint: kind-control-plane:6443

In the code above, we first identify the node we want to debug, then explicitly run kubectl debug using node/... as the parameter to access the node of our cluster. After that, when connected to the Pod, we use chroot /host to break through the chroot and fully enter the host. Finally, to verify that we can indeed see everything on the host, we check a portion of the kubeadm.conf, eventually seeing the configuration we set at the beginning of the article feature-gates: EphemeralContainers=true.

Conclusion

Being able to quickly and effectively debug applications and services can save a lot of time, but more importantly, it can greatly help resolve issues that could end up costing a lot of money if not addressed immediately. This is why tools like kubectl debug are so important to have at our disposal, even if they are not officially released or enabled by default. If enabling ephemeral containers is not an option, trying alternative debugging methods may be a good idea, such as using a debug version of the application image that includes troubleshooting tools; or temporarily changing the container command of the Pod to prevent it from crashing.

Original link:https://towardsdatascience.com/the-easiest-way-to-debug-kubernetes-workloads-ff2ff5e3cc75

END

The Easiest Way to Debug Kubernetes Workloads

Related posts:

Leave a Comment Cancel reply