Analyzing UDP Packet Loss Issues in Linux Systems

Recently, I encountered a UDP packet loss issue with a server application during work. While troubleshooting, I referred to a lot of materials. I primarily used tcpdump to capture packets at various stages where problems occurred, analyze where the issues arose, and troubleshoot accordingly. Eventually, I was able to resolve the problem. However, this situation is often due to issues within the service itself. If it’s an environmental issue, operating system, or even hardware, it may not be solvable from the service perspective. This article takes a different approach by analyzing potential packet loss causes from the external environment. After reading it, I found it very useful. Some sections have been modified from the original text, and I share it here for more people to reference.

Before we begin, let’s use an image to explain the process of receiving network packets in the Linux system.

1. First, network packets are sent to the network card via physical cables.

2. The network driver reads the packets from the network and places them in a ring buffer using DMA (Direct Memory Access), without CPU involvement.

3. The kernel reads the packets from the ring buffer for processing, executing the logic of the IP and TCP/UDP layers, and finally places the packets in the application’s socket buffer.

4. The application reads packets from the socket buffer for processing.

During the reception of UDP packets, any stage in the diagram can actively or passively drop packets; therefore, packet loss can occur at the network card and driver, or within the system and application.

The reason for not analyzing the sending data flow is that it is similar to the receiving process, just in the opposite direction. Additionally, the probability of packet loss during the sending process is lower than during receiving. Packet loss only occurs when the application’s sending rate exceeds the processing rate of the kernel and network card.

This article assumes the machine has only one interface named eth0. If there are multiple interfaces or the interface name is not eth0, please analyze according to the actual situation.

NOTE: In the text, RX (receive) indicates received packets, and TX (transmit) indicates sent packets.

Confirming UDP Packet Loss

To check if the network card has packet loss, you can use ethtool -S eth0 and look for fields corresponding to bad or drop. Under normal circumstances, the numbers corresponding to these fields should all be 0. If you see the corresponding numbers continuously increasing, it indicates that the network card has packet loss.

Another command to check for packet loss on the network card is ifconfig, which will output statistics for RX (received packets) and TX (transmitted packets):

[root@k8s-master ng]# ifconfig enp1enp2s0f1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500        ether 04:b0:e7:fa:75:9d  txqueuelen 1000  (Ethernet)        RX packets 0  bytes 0 (0.0 B)        RX errors 0  dropped 0  overruns 0  frame 0        TX packets 0  bytes 0 (0.0 B)        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0        device memory 0x92200000-922fffff

In addition, the Linux system also provides packet loss information for various network protocols. You can use netstat -s to view this information. By adding --udp, you can view only UDP-related packet data:

[root@k8s-master ng]# netstat -s -uIcmpMsg:    InType0: 17    InType3: 75    InType8: 77    OutType0: 77    OutType3: 692    OutType8: 249Udp:    5728807 packets received    12 packets to unknown port received.    0 packet receive errors    982710 packets sent    0 receive buffer errors    0 send buffer errorsUdpLite:IpExt:    InNoRoutes: 3    InBcastPkts: 497633    InOctets: 1044710406807    OutOctets: 17460621991142    InBcastOctets: 114600482    InNoECTPkts: 2886955071

For the above output, focus on the following information to check for UDP packet loss:

• packet receive errors is non-empty and continuously increasing, indicating UDP packet loss in the system.

• packets to unknown port received indicates that the received UDP packets are directed to a port where no application is listening, usually due to the service not being started, which does not cause serious issues.

• receive buffer errors indicates the number of packets dropped due to the UDP receive buffer being too small.

NOTE: A non-zero packet loss count does not necessarily indicate a problem. For UDP, a small amount of packet loss is often expected behavior, such as a packet loss rate (number of lost packets/number of received packets) of one in ten thousand or even lower.

Packet Loss at Network Card or Driver

As previously mentioned, if ethtool -S eth0 shows rx_***_errors, it is likely that there is a problem with the network card, leading to packet loss in the system. You need to contact the server or network card vendor for resolution.

[root@k8s-master ng]# ethtool -S enp1 | grep rx_ | grep errors     rx_crc_errors: 0     rx_missed_errors: 0     rx_long_length_errors: 0     rx_short_length_errors: 0     rx_align_errors: 0     rx_errors: 0     rx_length_errors: 0     rx_over_errors: 0     rx_frame_errors: 0     rx_fifo_errors: 0

netstat -i will also provide the number of packets received and transmitted, as well as the packet loss situation for each network card. Under normal circumstances, the output should show 0 for errors or drops.

If there are no issues with the hardware or driver, packet loss at the network card is typically due to the configured buffer size (ring buffer) being too small. You can use the ethtool command to view and set the network card’s ring buffer.

ethtool -g can be used to view the ring buffer of a specific network card. For example:

[root@k8s-master ng]# ethtool -g enp1Ring parameters for enp2s0f1:Pre-set maximums:RX:    4096RX Mini:  0RX Jumbo:  0TX:    4096Current hardware settings:RX:    256RX Mini:  0RX Jumbo:  0TX:    256

Pre-set indicates the maximum ring buffer value for the network card. You can set this value using ethtool -G eth0 rx 8192.

Packet Loss in Linux Systems

There are many reasons for packet loss in Linux systems, common ones include: UDP packet errors, firewalls, insufficient UDP buffer size, and high system load. Here, we analyze these causes of packet loss.

UDP Packet Errors

If UDP packets are modified during transmission, it can lead to checksum errors or length errors. Linux performs checks on UDP packets upon reception, and if an error is detected, the packet will be discarded.

If you want to ensure UDP packets with checksum errors are still sent to the application, you can disable UDP checksum checks through socket parameters.

Firewalls

If the system firewall causes packet loss, the behavior usually manifests as all UDP packets failing to be received. However, it is also possible that the firewall only drops a portion of the packets.

If you encounter a very high packet loss rate, please first check the firewall rules to ensure that the firewall is not actively dropping UDP packets.

Insufficient UDP Buffer Size

After receiving packets, Linux saves them in a buffer. Since the buffer size is limited, if UDP packets are too large (exceeding the buffer size or MTU size) or the rate of received packets is too fast, it can lead to packet loss due to the buffer being full.

On the system level, Linux has a maximum configurable receive buffer size, which can be checked in the following files. Typically, Linux sets an initial value based on the amount of memory at startup.

• /proc/sys/net/core/rmem_max: Maximum allowed receive buffer size

• /proc/sys/net/core/rmem_default: Default receive buffer size

• /proc/sys/net/core/wmem_max: Maximum allowed send buffer size

• /proc/sys/net/core/wmem_dafault: Default send buffer size

However, these initial values are not designed to handle large volumes of UDP packets. If the application receives and sends a large number of UDP packets, this value should be increased. You can use the sysctl command to make it effective immediately:

[root@k8s-master ~]# sysctl -w net.core.rmem_max=26214400 # Set to 25Mnet.core.rmem_max = 26214400

You can also modify the corresponding parameters in /etc/sysctl.conf to ensure they take effect at the next startup.

If packets are too large, you can split the data on the sender side to ensure each packet’s size is within the MTU.

Another configurable parameter is netdev_max_backlog, which indicates the number of packets that the Linux kernel can buffer after reading packets from the network card driver. The default is 1000, but this value can be increased, for example, to 2000:

[root@k8s-master ~]# sudo sysctl -w net.core.netdev_max_backlog=2000net.core.netdev_max_backlog = 2000[root@k8s-master ~]#

High System Load

High CPU, memory, or IO load can all lead to network packet loss. For example, if the CPU is under heavy load, the system may not have time to perform checksum calculations, memory copies, etc., resulting in packet loss at the network card or socket buffer. High memory load may cause the application to process packets slowly and fail to keep up, while high IO load may lead to the CPU being occupied with IO wait, leaving no time to process UDP packets in the buffer.

The Linux system is inherently interconnected; issues in one component can affect the normal operation of others. For high system load, either the application has problems, or the system is insufficient. For the former, timely detection, debugging, and fixing are required; for the latter, timely detection and scaling are necessary.

Application Packet Loss

As mentioned earlier, the UDP buffer size in the system can be adjusted through sysctl parameters, which only set the maximum allowed value. Each application must set its own socket buffer size when creating a socket.

Linux places received packets in the socket buffer, and the application continuously reads packets from the buffer. Therefore, two application-related factors can affect packet loss: the size of the socket buffer and the speed at which the application reads packets.

For the first issue, you can set the socket receive buffer size when initializing the socket in the application, for example, the following code sets the socket buffer to 20MB:

uint64_t receive_buf_size = 20*1024*1024;  //20 MBsetsockopt(socket_fd, SOL_SOCKET, SO_RCVBUF, &receive_buf_size, sizeof(receive_buf_size));

If you do not write and maintain the program yourself, modifying the application code may be difficult or even impossible. Many applications provide configuration parameters to adjust this value; please refer to the official documentation. If no configurable parameters are available, you may need to raise an issue with the program’s developers.

Clearly, increasing the application’s receive buffer will reduce the likelihood of packet loss, but it will also lead to increased memory usage, so caution is advised.

Another factor is the speed at which the application reads packets from the buffer. For applications, processing packets should be done asynchronously.

Where Packets are Lost

If you want to understand in detail which function the Linux system is losing packets in, you can use the dropwatch tool, which listens for system packet loss information and prints the function addresses where packet loss occurs:

# dropwatch -l kasInitalizing kallsyms dbdropwatch> startEnabling monitoring...Kernel monitoring activated.Issue Ctrl-C to stop monitoring1 drops at tcp_v4_do_rcv+cd (0xffffffff81799bad)10 drops at tcp_v4_rcv+80 (0xffffffff8179a620)1 drops at sk_stream_kill_queues+57 (0xffffffff81729ca7)4 drops at unix_release_sock+20e (0xffffffff817dc94e)1 drops at igmp_rcv+e1 (0xffffffff817b4c41)1 drops at igmp_rcv+e1 (0xffffffff817b4c41)

By using this information, you can find the corresponding kernel code and understand at which step the kernel discards packets and the general reasons for packet loss. During the troubleshooting process, I prefer to capture packets on various machines; this method is more suitable for tracking packet loss issues caused by the business itself, as shown below:

tcpdump -i network_interface_name udp port 2020 -s0 -XX -nn

Additionally, you can use the Linux perf tool to monitor the occurrence of the kfree_skb event (this function is called when network packets are discarded):

sudo perf record -g -a -e skb:kfree_skbsudo perf script

There are many articles online that reference the usage and interpretation of the perf command.

Summary

• UDP is inherently an unreliable protocol suited for scenarios where occasional packet loss does not affect the program state, such as video, audio, gaming, monitoring, etc. Applications requiring high packet reliability should not use UDP; it is recommended to use TCP instead. However, reliability can also be ensured at the application layer through retries and deduplication.

• If packet loss is detected on the server, first check the system load through monitoring. Try to reduce the load and then see if the packet loss issue disappears.

• If the system load is high, there are no effective solutions for UDP packet loss. If application issues cause high CPU, memory, or IO usage, promptly identify the abnormal application and fix it; if resources are insufficient, monitoring should promptly identify and facilitate scaling.

• For systems that receive or send a large number of UDP packets, adjusting the system and application’s socket buffer size can reduce the likelihood of packet loss.

• When processing UDP packets, applications should adopt an asynchronous approach and avoid excessive processing logic between receiving packets.

--END--

All articles from this public account have been organized into a directory. Please reply "m" in the public account to get it!
Recommended reading:
Programmers are disappearing.

You must have heard these not-so-standard pronunciations in the tech circle...

The number of readers has surpassed 100,000! (Exclusive benefits must not be missed!)

5T technical resources are being released! Including but not limited to: C/C++, Linux, Python, Java, PHP, Artificial Intelligence, Microcontrollers, Raspberry Pi, etc. Reply "1024" in the public account to get it for free!!