Layered Troubleshooting Method for Embedded Linux Network Issues

Hello everyone, I am the Intelligence Guy~

In embedded Linux development, network issues are often “invisible killers”—sudden disconnections, inaccessible services, data packet loss… These may involve multiple aspects such as hardware, drivers, protocol stacks, and firewalls. Today, I will share how to troubleshoot network issues in embedded Linux, guiding you through the ultimate troubleshooting process from the physical layer to the application layer, also known as the layered troubleshooting method:

1. Physical Layer Troubleshooting: Is the Network Card “Alive”?

Typical Issues

• The network card is not recognized, the network cable is not properly connected, or the physical link is broken.• Abnormal network card driver (e.g., packet loss, CRC errors).

Tools and Commands

1. <span>ifconfig</span> / <span>ip addr</span>

   ifconfig eth0        # Check the network card status (UP/RUNNING)
   ip addr show eth0    # More modern alternative command

   • Key Fields：◦ <span>RX/TX packets</span>：Packet statistics, if 0, the link may be down.◦ <span>errors</span>/<span>dropped</span>：Driver or hardware error counts.

2. <span>dmesg</span>

   dmesg | grep eth0   # Check the network card initialization information in the kernel log

   • Check if the driver loaded successfully and if there are any errors (e.g., <span>link down</span>).

3. <span>/sys/class/net</span>（Alternative when no tools are available）

   cat /sys/class/net/eth0/operstate  # Displays "up" or "down"
   cat /sys/class/net/eth0/carrier    # Whether the physical link is active (1=normal)

2. Network Layer Troubleshooting: Is IP Connectivity Normal?

Typical Issues

• Incorrect IP address configuration (static/DHCP failure).• Subnet mask or gateway errors, leading to cross-segment connectivity issues.

Tools and Commands

1. <span>ping</span>

   ping 192.168.1.1       # Test connectivity to the gateway
   ping -c 4 8.8.8.8     # Limit to sending 4 packets (to avoid hanging)

   • If the gateway is reachable but the external network is not, it may be a DNS or routing issue.

2. <span>udhcpc</span>（DHCP Client）

   udhcpc -i eth0 -v      # Dynamically obtain IP (commonly used in embedded systems)

   • If unable to obtain an IP, check the DHCP server or network cable.

3. <span>route</span> / <span>ip route</span>

   route -n               # View the routing table
   ip route show default  # Show the default gateway

   • Ensure the default gateway (<span>default</span>) points to the correct router IP.

3. Transport Layer Troubleshooting: Is the Port Listening?

Typical Issues

• Service not started, port blocked by firewall.• Port binding conflicts (e.g., multiple processes occupying the same port).

Tools and Commands

1. <span>netstat</span> / <span>ss</span>

   netstat -tuln          # View all listening ports (old BusyBox version)
   ss -lntp               # More efficient tool (requires cross-compilation support)

   • Confirm if the service is listening on the expected port (e.g., <span>0.0.0.0:80</span>).

2. <span>telnet</span> / <span>nc</span>

   telnet 192.168.1.100 80  # Test TCP port connectivity
   nc -uz 192.168.1.100 53  # Test UDP port (requires nc support)

   • If local access works but external does not, it may be blocked by a firewall.

4. Firewall and Routing: Is Traffic Being Blocked?

Typical Issues

• iptables/nftables rules dropping traffic.• NAT configuration errors (port forwarding failures).

Tools and Commands

1. <span>iptables</span>

   iptables -L -nv --line-numbers  # View all rules
   iptables -t nat -L              # View NAT table

   • Focus on checking if there are <span>DROP</span> rules in the <span>INPUT</span> and <span>FORWARD</span> chains.

2. <span>/proc/net/ip_conntrack</span>（Connection Tracking）

   cat /proc/net/ip_conntrack      # View active NAT connections

   • Confirm if NAT sessions are established normally (e.g., TCP/UDP entries).

5. Packet Capture Analysis: The Ultimate Weapon for Locating Protocol Issues

Typical Scenarios

• Packets not reaching the device, protocol format errors, application layer packet loss.

Tools and Commands

1. <span>tcpdump</span>（Embedded Simplified Version）

   tcpdump -i eth0 -nn port 80   # Capture traffic on port 80
   tcpdump -i eth0 icmp          # Capture PING packets

   • If no packets are received, it may be a switch/VLAN configuration issue.

2. Wireshark Analysis• Import the capture file (<span>.pcap</span>) into Wireshark on a PC to analyze protocol interaction details.

6. Special Handling in Resource-Constrained Environments

1. Use<span>/proc/net</span> to Count Packet Loss

   cat /proc/net/dev          # View packet statistics for each network card
   cat /proc/net/snmp         # View packet loss at IP/TCP/UDP layers

2. Static Compilation Toolchain• Cross-compile static versions of <span>tcpdump</span>, <span>strace</span>, etc., and place them in the embedded device.

Summary: Quick Reference for Embedded Network Troubleshooting

Issue Level	Troubleshooting Tools	Key Commands
Physical Layer	ifconfig, dmesg	<span>ifconfig eth0</span>, <span>dmesg</span>
Network Layer	ping, route	<span>ping 8.8.8.8</span>, <span>ip route</span>
Transport Layer	netstat, telnet	<span>netstat -tuln</span>, <span>telnet</span>
Firewall	iptables	<span>iptables -L -nv</span>
Protocol Analysis	tcpdump	<span>tcpdump -i eth0 port 80</span>

Therefore, the core of troubleshooting network issues is tolayer down the scope, verifying step by step from physical cables to application protocols. In embedded scenarios, effectively utilizing built-in system tools and the <span>/proc</span> filesystem can quickly locate issues even without advanced tools!

END

Author:Mr. Deng

Source:Embedded Intelligence BureauCopyright belongs to the original author, please contact for deletion if there is any infringement.▍Recommended ReadingOverview of Job Positions and Salary Status at ZhiHuiJun CompanyA Guide to Writing Embedded Driver Programs (Based on Timing Diagrams)Why is Hardware Harder than Software, but Hardware Engineers Earn Less?→ Follow to avoid getting lost ←