100G Network Rate Optimization – NIC Device Driver Tuning

NIC Device Driver Tuning

NIC Device Driver

Typically, the device driver versions included with Linux distributions are outdated and may not support all the latest features. This page provides information on updating device drivers.

To check the current driver version, execute the following command:

ethtool -i ethN

The command ethtool -i ethN is used in Linux systems to query the driver information for a specified network interface (ethN). This command outputs key information such as the driver name, version, firmware version, and bus location, helping users diagnose driver compatibility or hardware status issues. For example: ethtool -i eth0

mlx5_core Driver (NVIDIA/Mellanox) Update

The latest drivers can be obtained from the following URL: https://developer.nvidia.com/doca-downloads

  • Here is the link for the updated driver suitable for Ubuntu 22.04:

https://developer.nvidia.com/doca-downloads?deployment_platform=Host – Server&deployment_package=DOCA – Host&target_os=Linux&Architecture=x86_64&Profile=doca – ofed&Distribution=Ubuntu&version=22.04&installer_type=deb_local

New driver features worth trying (only for ConnectX-7):

The DOCA_v2.10.0 version (released in February 2025) of NVIDIA’s “doca-ofed” driver now supports hardware Generic Receive Offload (GRO) (previous versions of this driver only supported the sender). As stated in the release notes: “Hardware Generic Receive Offload (GRO) reduces CPU overhead and increases throughput by offloading the reassembly task of incoming network packets from the CPU.”

To enable this setting, execute the following command:

/usr/sbin/ethtool -K ethN rx-gro-hw on

Please note that this feature is only effective when the ring buffer size does not exceed 2K; in fact, a default ring buffer size of 1024 seems optimal in this version of the driver.

After enabling rx-gro-hw, the table below shows the throughput improvement between two ESnet test hosts in the local area network. The wide area network throughput is unaffected as the bottleneck is at the sender.

RX GRO HW setting 1500B MTU 9000B MTU
off (default) 20 Gbps 41 Gbps
on 36 Gbps 63 Gbps

Note that this version of the driver displays the following when executing the ethtool -i command:

driver: mlx5_core
version: 25.01-0.6.0 

You also need to ensure that the firmware is up to date. The mlxup tool can easily accomplish this.

Other Drivers:

If you have details about other drivers, please email us the information, and we will add it here.

Source

https://fasterdata.es.net/host-tuning/linux/100g-tuning/nic-device-driver/

Leave a Comment