In nearly a decade of our involvement in cloud-native network development, we have deeply realized that troubleshooting network issues, even in small environments, can turn into a nightmare. Much of the complexity arises from the Linux kernel itself—sending a single packet can involve dozens of kernel functions from different subsystems. Traditional tools like tcpdump often fall short due to their typically coarse granularity.

This frustration during the debugging process led us to create a networking debugging tool based on eBPF—”pwru” (which stands for “packet, where are you?”). Initially, it was focused solely on the host network stack. However, as eBPF-based networking solutions have become more prevalent, more and more host network functionalities have been migrated into BPF programs. Therefore, we set out to extend this tool to support tracing BPF programs.

In this presentation, we will first introduce the specific implementation of this debugger. Next, we will discuss the efforts we made to bypass some limitations of the BPF subsystem while expanding the tool. Finally, we will showcase several interesting cases of network issues successfully debugged using this tool.

This article is translated from Linux Plumbers Conference 2024 (September 18-20, 2024): pwru – Linux kernel and BPF-based networking debugger · Indico^[1]

pwru – A Networking Debugger Based on Linux Kernel and BPF

Presentation video (bilingual subtitles)

Introduction: A Tool Born from the Pain of Network Debugging

Hello everyone, I am Martynas, and this is Gray. We are both data path engineers at Isovalent (now part of Cisco). As data path engineers, we often think we spend 80% of our time writing code, but the reality is that we spend over 80% of our time troubleshooting. To alleviate this pain, we ultimately built a debugging tool, which is the main topic of our discussion today.

This presentation itself is a great example of “eating our own dog food” because we will use BPF to troubleshoot issues in the BPF data plane. During this process, we encountered some tricky limitations and had to use some creative methods to solve them, which we will discuss later.

1. Why Do We Need a New Networking Debugging Tool?

In the world of Linux container networking, we face quite a few layers of virtualization, such as namespaces, virtual network devices, and various connections between them. Even with an eBPF-based data plane, we still rely on the standard Linux network protocol stack.

If you zoom in on the Linux penguin icon, you will see hundreds of kernel functions involved in packet processing and routing. This image comes from the Linux Foundation’s wiki page, and I remember it was based on the 2.6 kernel version. Since then, things have changed significantly, and the network stack has become more complex.

Therefore, searching for a problematic network packet in a complex path is a bit like playing a game of “Where’s Waldo?” The difference is that when looking for Waldo, we have reliable tools—our eyes. In contrast, the tools at our disposal for troubleshooting network packets are relatively limited.

The most obvious method is to keep adding debugging print statements, but this is unreliable as it greatly increases our feedback loop and latency in discovering issues. This process becomes even worse in environments where we cannot directly SSH in: you add debugging statements, show them to the customer, ask them to run it, the customer provides you with a new trace log, and then you have to start over.

Here are a few existing solutions:

1. tcpdump: This is a commonly used tool, but its granularity is too coarse. Many things can happen between two capture points in tcpdump.
2. bpftrace: We also use it frequently. Essentially, we mount a kprobe on the kfree_skb function and then obtain the kernel call stack that led to the execution of that function. However, we found that filtering this way can be cumbersome. While you can access all parameters (in this case, SKB), writing TCP-level filters becomes very tedious, and it also requires a compiler.
3. ipftrace2: This is another tool developed by our colleague Yutaro. It does similar things to what we do, but it relies on the mark field of SKB. In the BPF data plane, we have extensively used the mark field to pass state between different function executions, so the mark field can no longer be used for tracing, making ipftrace2 unfeasible in our scenario.

2. Introduction to pwru: Find Your Lost Packets

It was this frustration in debugging that prompted us to build our own tool. It is called “Packet, where are you?” pronounced as pwru, and we have open-sourced it.

From the output provided by pwru, you can see that you can use tcpdump-style filters. Then, you can obtain all the metadata you are interested in for troubleshooting. In this example, we have an iptables rule dropping certain traffic, and through the trace logs, we can clearly see what happened to the packet we filtered.

The core idea of this tool’s simplified version is: we parse the BTF (BPF Type Format) of the kernel or kernel modules to find all functions that accept SKB (Socket Buffer) as parameters. Then, we attach BPF filtering programs to these functions using fentry, kprobe, or kprobe_multi. These programs push events into a queue, and then we read these events from user space—it’s a very straightforward principle.

pwru has a very flexible deployment method:

• Standalone Binary: You can run pwru as a statically linked binary with no external dependencies, under 10MB in size, making it very easy to distribute to your customers.
• Docker Container: You can also run it in a standalone Docker container, but it needs to be in privileged mode.
• Kubernetes Pod: For us Cilium developers, our target platform is Kubernetes, so we can run it as a Kubernetes Pod and decide on which nodes to execute it.
• GitHub Actions: We often use it to debug intermittent issues (flakes) in CI (Continuous Integration).
• Local Multi-node Environment: Another use case I haven’t listed is when you want to run locally in a multi-cluster environment; we can do this on our own laptops. By running multi-node Cilium in a special way in kind (Docker in Docker) containers, since everything is still running on your host, you can use pwru to see the cluster view of packets and trace their paths across all nodes.

Next, I will hand over to Gray, who implemented some advanced features of this tool. The initial version was very inflexible; we only had some fixed maps to add filters for IP, port, protocol, etc. But soon we encountered limitations with this filtering method, so Gray started working on solving this problem.

3. Core Implementation: Dynamic Filters and BPF Program Injection

Thank you. Next, let’s talk about how to implement tcpdump-like filters.

Overall, this is a very simple three-step process:

1. Use functions from the libpcap library to compile the filter expression into cBPF.
2. Use the cloudflare/cbpfc library to convert cBPF into eBPF.
3. Inject the generated eBPF into pwru.

Generally, to run a BPF program, the kernel first compiles the BPF C code into a BPF ELF object. Then, a library reads this BPF object into user space memory. Finally, we load this BPF object into the kernel. There is a timing point in this process where we can perform the injection: after the BPF object is read into user space memory but before it is loaded into the kernel. This is the moment we inject the BPF code.

Next, let’s look at some details.

Step 1: Compile the Expression into cBPF

We just need to call functions in libpcap. We can easily do this in Go using cgo: specify CFLAGS and LDFLAGS to locate libpcap, include the header files, convert Go strings to C strings, initialize a cBPF buffer, and then call the libpcap‘s pcap_compile function. This way, we obtain the operation codes for cBPF. The key point here is that we must compile libpcap ourselves because we need to disable dbus, which is the only way to statically link libpcap.

Step 2: Convert cBPF to eBPF

We use Cloudflare’s cbpfc library. The ToEBPF function takes cBPF and returns eBPF. For example, the cBPF expression compiled in the first step can be converted to eBPF through this API.

However, there is a problem: the converted eBPF uses direct packet access. The kernel’s verifier will reject this generated eBPF because only a few types of programs are allowed to use direct access. To solve this problem, pwru must scan all converted eBPF, find all instructions with direct packet access, and convert them to indirect access through calls to eBPF helper functions. Specifically, we must call the bpf_probe_read_kernel eBPF helper function.

Here is an example. Suppose we want to read the value pointed to by the R4 pointer into the R0 register. We must first read this value onto the stack, and then move it from the stack to R0. To do this, we need to set R1 to the stack address (R10 – 8), R2 to the data size (1 byte), and R3 to the data source (R4). Then we call the helper function to get the data onto the stack. Finally, we can move the value from the stack to the R0 register.

But this is not enough because we used additional registers R1 to R3 here. The original instruction only used R4 and R0, but after conversion, we used three more registers, which may already be occupied. To resolve this conflict, we must save the values of R1 to R3 onto the stack before conversion and restore them from the stack after conversion. Thus, a direct access instruction is ultimately converted into a code block: save registers to stack -> call eBPF helper function -> restore registers from stack.

Step 3: Inject the Generated eBPF into pwru

pwru needs a stub function, such as filter_pcap_ebpf_l3. The first two parameters are SKB pointers, but they are not actually used. I mean, they are used in the original definition of this function, but the entire function body will be replaced by the eBPF code we generate. They exist because we need at least three available registers to complete the conversion. So, with them, we have R1, R2, R3 as available registers, plus R0 for the return value. The fourth and fifth parameters are skb->data and skb->data_end, which are necessary for the conversion process. In the second step, we must pass R4 and R5 as eBPF options for correct conversion.

Then, pwru will scan the original BPF operation codes, find the symbol of this stub function, and replace its function body with the code we generated in the second step. Finally, this new BPF operation code is loaded into the kernel. This is how we implement dynamic tcpdump-style filters in pwru.

4. Advanced Features: In-depth Tracing of BPF Programs

Next, we discuss tracing BPF programs. What does this mean? Let’s look at a typical pwru output. It shows a packet being received by __netif_rx, then processed by TC BPF (TC is the identifier for TC BPF programs). Next, it is redirected by BPF to another network interface eth0. I want to emphasize a fact: this packet underwent SNAT, changing the source IP from 10.244.3.106 to 172.21.0.2.

My question is: which part of the code in the TC eBPF program is responsible for this SNAT? Looking at this diagram, it shows the tail call jump relationships of eBPF programs in Cilium. Which part of the eBPF program executed the SNAT? We have no way of knowing; pwru does not tell us.

To solve this problem, the first natural idea is to use kprobe to attach another BPF program. We tried to attach a kprobe to the TC eBPF program. Unfortunately, the kernel does not allow this. The kernel function check_kprobe_address_safe will throw an error if the target address of the kprobe is another BPF program. So, kprobe does not work.

However, we can use fentry to attach another BPF program, and the kernel does indeed load it. But for tail call tracing, this is not useful. Because the way fentry attaches is by replacing five bytes of instructions in the prologue of the BPF program. When a BPF program runs, bpf_trampoline is executed first, which is how we observe the execution of BPF programs. But for tail calls, such as program A tail calling program B, the prologue of program B is directly skipped by the tail call jump. Therefore, while fentry can be attached to BPF programs, it does not help with tail call tracing because the trampoline is skipped.

pwru’s solution is: use kprobe to attach to all BPF helper functions.

pwru first scans /proc/kcore to collect all existing BPF helper functions in the kernel. Then, pwru attaches a BPF program to all these helper functions. This program does the following:

1. Unwind the stack to calculate a stack id. If you are familiar with BPF helper functions, there is already a function in the kernel called bpf_get_stackid. Here is a simplified version of the implementation; we unwind the stack to the root and use some information from the stack as the stack id. This way, all functions on the same call chain will have the same stack id.
2. Read the SKB pointer from a BPF map using the stack id. This SKB pointer is stored by other functions that executed before the BPF helper function on the same call chain. Since they are on the same call chain, the stack id is the same, and we are confident we can retrieve the correct SKB pointer.
3. Collect the information we need, such as the pointer to the network device interface. But the key point here is that we also collect the caller program counter (PC) of this event.
4. This caller PC can be converted into a symbol using the /proc/kallsyms file. This symbol provides the name information of the BPF program, even for tail-called programs.

This is an example. Originally, we had such pwru output that only showed the most basic information about BPF execution. By enabling BPF helper function tracing, we can see the newly added caller information, showing the name of the eBPF program. It shows that the packet was received and then entered the TC eBPF program. The first BPF helper function called is bpf_skb_event_output, which is called by the BPF program cil_from_container. This is the first BPF program. Then, we tail call to the second BPF program tail_handle_ipv4. After a while, the third program tail_handle_ipv4_cont is called. Finally, the last program tail_nodeport_rev_dnat_ingress_ipv4 is tail called.

By observing the tuple information of SKB, we can clearly see that SNAT occurs here, in the tail_nodeport_rev_dnat_ingress_ipv4 program. Therefore, by enabling this feature, we can gain a detailed understanding of the internal workings of BPF programs.

Future Prospects

What we see now is just a rough tracing of BPF programs. If we could collect and parse parameters, especially for BPF map function parameters, it would be even more helpful. As long as we have the BTF type information for map keys and map values, this is feasible. We just need to call the bpf_snprintf_btf() helper function, and the kernel can parse the types and generate strings for us, which is very convenient.

Another thing I think would be very helpful is outputting source code. This is a typical output of the bpftool prog dump command. As you can see, the source code information is already in the kernel because Cilium does not strip debugging information from ELF files. Therefore, all debugging symbols exist in the kernel’s BPF objects. We just need to associate this source code information with the calls to BPF helper functions and print it out. This way, we can know which line of source code corresponds to the call of a certain BPF helper function. This would be very helpful for developers debugging.

5. More Powerful Tracing Capabilities

Next, we discuss XDP tracing and SKB tracing, some features we have mentioned before.

1. XDP Tracing

By enabling the --filter-trace-xdp flag, we can also trace XDP. As shown here, this is a virtual XDP program. This feature is implemented by using fentry to attach to all XDP eBPF programs and applying the BPF filter injection mechanism we introduced earlier. This way, we can filter at the XDP level.

2. SKB Tracing (Default Mode)

I want to emphasize three points here:

• --filter-track-skb: As we mentioned, pwru uses tcpdump filters to filter SKB. By enabling this flag, we can trace SKB by pointer address. This feature is very useful when network encapsulation or encryption occurs. For example, the original packet may match the pcap filter, but after a while, this packet may be encrypted. After encryption, we can no longer match the filter because all payloads are encrypted. But by enabling this flag, we can still trace this SKB because its pointer address remains unchanged. This is a very convenient way to understand the lifecycle of SKB.
• --filter-non-skb-funcs: As Martynas mentioned earlier, pwru was initially implemented by tracing kernel functions that included SKB pointers as parameters. But by enabling this flag, we can also trace any kernel function, even if it does not have SKB pointers as parameters. This is also achieved through the stack id we introduced earlier. We collect the stack id, so all functions on the same call chain have the same stack id. This way, these functions can retrieve the SKB pointer from the BPF map and collect information from that SKB. In this way, even non-SKB functions can be probed. An example of using this is tracing the xfrm state lookup functions. There are four xfrm state lookup functions in the kernel, none of which are SKB functions. Now we can trace these xfrm states to see if they are called and when they are called.
• Handling Rebuilt SKB: In some special cases, SKB will be rebuilt. For example, when an SKB crosses a network namespace via veth, and an XDP program is attached to this veth. The way the kernel handles this is by consuming the original SKB and then rebuilding a new SKB, but the new SKB has exactly the same network header and payload. This is a problem for us because this rebuilt SKB will cause us to lose track of the SKB, even if the SKB tracking feature is enabled, as this is a new SKB with a changed pointer address. But now, pwru can understand this rebuilding behavior, so you don’t have to worry about all these special cases. pwru can continue to trace the new SKB after the rebuild. This is very useful.

Well, that’s all I wanted to share. Thank you all!

Reference Links

[1] Linux Plumbers Conference 2024 (September 18-20, 2024): pwru – Linux kernel and BPF-based networking debugger · Indico: https://lpc.events/event/18/contributions/1942/