0. Introduction: Why You Need “Tools” Instead of “Common Commands”
Sometimes, strange lags cling to the screen like fog: latency fluctuates, and packet loss appears intermittently, as if the business is being choked by an invisible hand. Don’t rush to blame the “network black hole,” and don’t immediately restart everything. Treat the system like an instrument to tune: which string is loose, which part is resonating, and let the evidence speak. The following troubleshooting paths and operation checklists can be applied directly; each step provides commands and simulated outputs to closely replicate the scene.
1. Locating the Problem: Breaking Down Complex Issues into Four Questions
- • Is this an endpoint issue or a middle path issue? (Host stack/NIC vs. Routing/Peer)
- • Is it a momentary congestion or a persistent bottleneck? (Peak vs. Steady State)
- • Is it L2/L3 physical/MTU/packet loss, or L4/L7 retransmission/queuing?
- • What is the scope of the impact? (Same data center, cross-region, public network)
Clear judgment leads to a sequence of actions.
2. Ten-Minute “Quick Check” Process (No Configuration Changes, Just Observation)
1) Measure End-to-End Latency Breakdown (Connection, TLS, First Packet, Total Time)
$ curl -sS -o /dev/null -w "code=%{http_code} conn=%{time_connect}s tls=%{time_appconnect}s ttfb=%{time_starttransfer}s total=%{time_total}s\n" https://api.example.com/ping
code=200 conn=0.012s tls=0.034s ttfb=0.078s total=0.095sInterpretation: If conn and tls are very small, but ttfb suddenly increases, it is more likely due to slow server/upstream processing; conversely, if conn occasionally increases, it seems like the network path is acting up.
2) Observe Real-Time RTT, Retransmissions, and Congestion Window from TCP Perspective
$ ss -ti '( dport = :443 or sport = :443 )' | head -n 8
ESTAB 0 0 10.0.0.5:52364 203.0.113.10:443
cubic wscale:7,7 rto:204 rtt:78.2/2.1 ato:40 mss:1448 cwnd:10 retrans:1/19Interpretation: rtt:78.2/2.1 indicates average/jitter; retrans:1/19 means there was 1 retransmission out of 19 segments—slightly acceptable, but a continuous increase should be monitored.
3) End-to-End Path Health (MTR to Observe Latency and Packet Loss)
$ mtr -ezbw -c 100 api.example.com
Start: 2025-08-10T10:30:00
HOST: app-01
1.|-- 10.0.0.1 0.2% 100 0.3 0.5 0.2 2.1
2.|-- 10.0.3.254 0.0% 100 0.7 0.9 0.5 3.0
3.|-- 172.16.1.1 5.0% 100 2.1 5.6 1.9 20.4
4.|-- 203.0.113.10 5.0% 100 6.9 12.2 5.7 42.7Interpretation: The 3rd and 4th hops have 5% packet loss, which may indicate congestion or rate limiting starting at the 3rd hop; if only the intermediate nodes have packet loss but the endpoint is 0%, it may just be that the intermediate device is rate limiting ICMP, so there is no need to panic.
4) Check if the Local Machine is Experiencing Packet Loss (Driver/Queue/Ring Buffer)
$ ip -s link show dev eth0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...
RX: bytes packets errors dropped overrun ...
8.4G 9.2M 0 5321 0
TX: bytes packets errors dropped carrier ...
7.1G 8.1M 0 0 0RX dropped continuously increasing is mostly due to receive buffer overflow/soft interrupts not keeping up.
$ ethtool -S eth0 | egrep -i 'drop|err|rx_.*no|fifo' | head
rx_no_buffer: 4873
rx_missed_errors: 0
rx_fifo_errors: 05) Verify MTU / Path MTU Black Hole
$ tracepath -n api.example.com | head -n 3
1?: [LOCALHOST] pmtu 1500
1: 10.0.0.1
2: 10.0.3.254 pmtu 1500$ ping -c3 -M do -s 1472 api.example.com
PING ... 1472(1500) bytes of data.
3 packets transmitted, 3 received, 0% packet loss, time 2004msIf 1472 succeeds but larger packets fail, it indicates PMTU is normal; if even 1472 cannot pass while smaller packets do, suspect that intermediate devices are dropping “non-fragmentable” ICMP—typical PMTUD black hole.
3. System-Level Evidence: Turning “Feels Slow” into “Digital Fingerprints”
1) Global Count of Packet Loss and Retransmissions
$ nstat -az | egrep 'TcpRetransSegs|IpInDiscard|IpInDelivers|UdpInErrors'
TcpRetransSegs 321
IpInDiscard 57
UdpInErrors 0$ sar -n DEV,EDEV 1 3
11:20:01 AM IFACE rxpck/s txpck/s rxerr/s txerr/s rxdrop/s txdrop/s
11:20:02 AM eth0 15234 13012 0.00 0.00 5.00 0.00rxdrop/s continuously not being 0 indicates packet loss on the host side.
2) Soft Interrupt and ksoftirqd Pressure
$ top -H -b -n1 | sed -n '1,15p'
%Cpu(s): 12.3 us, 7.8 sy, 0.0 ni, 60.0 id, 0.0 wa, 19.7 si, 0.2 stsi (softirq) being high + ksoftirqd/* threads being prominent indicates interrupt storm/high traffic preempting CPU.
$ cat /proc/softirqs | sed -n '1,12p'
CPU0 CPU1
NET_RX: 1234567 345678
NET_TX: 234567 1234563) Clues from Queues, qdisc, and Bufferbloat
$ tc qdisc show dev eth0
qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap ...pfifo_fast is prone to queue bloat → high latency in burst scenarios. The method to switch to fq_codel will be provided later.
4. Common Root Causes and Remedial Actions (with “How to Verify”)
The following adjustments should be validated in a non-production environment first, and then implemented according to change processes. For modifications involving the kernel network stack, it is recommended to first test on a single machine, then expand the scope.
A. Receive Ring/Soft Interrupt Handling Overload → Adjust NIC Ring/RPS/Interrupt Affinity
1) Check and moderately increase the ring buffer (if hardware/driver allows):
$ ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums: RX: 4096 RX Mini: 0 RX Jumbo: 0 TX: 4096
Current hardware settings: RX: 512 TX: 512
$ sudo ethtool -G eth0 rx 2048 tx 10242) Spread soft interrupts across multiple cores (RPS/RFS example):
# Set RPS mask for eth0's queue (example mask 0x3 -> use CPU0/CPU1)
$ echo 3 | sudo tee /sys/class/net/eth0/queues/rx-0/rps_cpus3) Verification: sar -n EDEV 1‘s rxdrop/s should decrease, and ss -ti‘s retrans should stabilize.
B. MTU/PMTUD Black Hole → Unify MTU or MSS Clamp at Boundaries
1) Identify abnormal paths (large packets not passing):
$ ping -c3 -M do -s 1472 api.example.com # 1500 test
$ ping -c3 -M do -s 8972 api-internal.example.com # If internal is 9000 jumbo2) Temporary mitigation (clamp MSS to path MTU at the gateway):
# iptables version
$ sudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
# nftables version (recommended for new systems)
$ sudo nft add rule inet filter forward tcp flags syn tcp option maxseg size set clamp-to-pmtu3) Long-term fix: unify subnet MTU (common in cloud environments with mixed 1500/9001), or ensure ICMP “needs fragmentation but DF set” returns.
4) Verification: tracepath PMTU stabilizes, and large packet business transmission no longer times out.
C. Bufferbloat (Queue Bloat) → Switch to FQ-CoDel / Enable BBR
1) Switch the default queue to fq_codel (reduce queuing latency):
$ echo 'net.core.default_qdisc=fq_codel' | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -p
$ tc qdisc replace dev eth0 root fq_codel2) Enable a more friendly congestion control (BBR, must be compatible with both internal and external networks):
$ echo 'net.ipv4.tcp_congestion_control=bbr' | sudo tee -a /etc/sysctl.conf
$ sudo sysctl -p
$ sysctl net.ipv4.tcp_available_congestion_control
net.ipv4.tcp_available_congestion_control = cubic reno bbr3) Verification: Under long flows, ss -ti‘s rtt jitter converges, tc -s qdisc‘s drops no longer continuously increase; end-to-end curl -w total latency decreases.
D. CPU Not Keeping Up (Soft Interrupts and User Space Contention) → Interrupt Affinity/Multi-Queue/NUMA Binding
1) Bind NIC queue interrupts to different CPUs to avoid concentrating on CPU0:
$ grep eth0-TxRx /proc/interrupts
57: 123456 IR-PCI-MSI eth0-TxRx-0
58: 98765 IR-PCI-MSI eth0-TxRx-1
# Bind interrupt 57 to CPU0, 58 to CPU1
$ echo 1 | sudo tee /proc/irq/57/smp_affinity_list
$ echo 2 | sudo tee /proc/irq/58/smp_affinity_list2) Enable RSS/multi-queue receive/transmit (if hardware supports):
$ ethtool -l eth0
Channel parameters for eth0:
Pre-set maximums: RX: 8 TX: 8
Current hardware settings: RX: 2 TX: 2
$ sudo ethtool -L eth0 combined 43) Verification: /proc/interrupts shows balanced distribution, si% decreases, and rxdrop/s converges.
E. Firewall/ACL/Rate Limiting Causing Intermittent ICMP or Port Packet Loss → Use TCP MTR and Counter Evidence
1) When ICMP is rate-limited, use TCP probing:
$ mtr -ezbw -c 50 -T -P 443 api.example.com2) Check local firewall counters (which rules are consuming packets):
# iptables
$ sudo iptables -L -v -n | head -n 20
Chain INPUT ...
pkts bytes target prot opt in out source destination
# nftables
$ sudo nft list ruleset | sed -n '1,80p'3) Verification: After relaxing or correcting rules, curl -w and mtr packet loss returns to normal.
F. Slow Server Processing Misjudged as “Network Slow” → Split Time from TCP Layer
1) Observe normal RTT but high ttfb, indicating the network is not the main cause:
$ ss -ti state established '( dport = :443 )' | head -n 6
ESTAB ... rtt:8.1/1.2 retrans:0 ...2) Confirm “cold hotspots” on the server side: disk IO, upstream dependencies, thread pools. After excluding the network, refocus on the application.
G. UDP Jitter/Packet Loss (Real-Time Audio/Video/Log Delivery) → Use iperf3 in UDP Mode
# Target Side
$ iperf3 -s
# Source Side
$ iperf3 -u -b 200M -l 1200 -c 203.0.113.10 -t 30
[ 5] 0.00-30.00 sec 690 MBytes 193 Mbits/sec 1.2% 3.45 ms 4.12 ms 12.1 msJitter and packet loss rates are visually presented; lower the rate/fragment length to observe threshold points.
5. Integrated Application Verification: Don’t Just Look at the Network, Be Responsible for the Business
1) HTTP Health Check and 95/99 Percentile Latency
$ for i in {1..50}; do curl -s -o /dev/null -w "%{time_total}\n" https://api.example.com/ping; done | awk '
{p[NR]=$1} END{
n=asort(p);
p95=p[int(0.95*n)]; p99=p[int(0.99*n)];
printf("p95=%.3fs p99=%.3fs\n", p95, p99)
}'
p95=0.112s p99=0.148s2) TCP Connection Failure Rate (Perceptible by Applications)
$ journalctl -u myapp -n 200 --no-pager | egrep -i 'ECONNRESET|ETIMEDOUT|EHOSTUNREACH' | wc -l
3Incorporating “connection error count/request count” into monitoring is more reliable than visually tracking logs.
6. Common “Pitfall Gallery” (You Will Likely Encounter)
- • Mixed MTU in Cloud Environments: 1500 within VPC, 9001 across AZs/dedicated lines, leading to packet loss across boundaries.
- • Inconsistent MTU between Containers and Hosts: CNI defaults to 1450, host is 1500, external GRE/VXLAN overlay—use
tracepathto clarify. - • Driver Bugs/Outdated Firmware:
dmesg | grep -i 'NETDEV WATCHDOG',link down/upjitter; upgrading the kernel/driver is the most stable solution. - • Default Queue pfifo_fast: RTT explodes during peaks; switching to
fq_codeloften yields immediate results. - • ICMP Completely Blocked:
pingnot passing ≠ service not passing; remember to usemtr -Tandcurl -was dual evidence. - • conntrack Being Saturated: Instantaneous connection spikes,
nf_conntrack_countnearingmax, packet loss disappears like a receding tide; adjusting limits/tuning timeouts/load balancing is the remedy.
$ cat /proc/sys/net/netfilter/nf_conntrack_count
1048575
$ cat /proc/sys/net/netfilter/nf_conntrack_max
10485767. Drill Script (Non-Intrusive, Read-Only Sampling) — Run Once to Capture Key Evidence
Only collect, do not change the system; can be executed on a bastion host or business machine. Please save outputs according to security specifications.
#!/usr/bin/env bash
# Name: net-quickcheck.sh (Read-Only Health Check)
set -euo pipefail
TARGET="${1:-api.example.com}"
PORT="${2:-443}"
COUNT="${3:-100}"
echo"=== curl timing (${TARGET}) ==="
curl -sS -o /dev/null -w "code=%{http_code} conn=%{time_connect}s tls=%{time_appconnect}s ttfb=%{time_starttransfer}s total=%{time_total}s\n""https://${TARGET}:${PORT}" || true
echo -e "\n=== TCP sockets RTT/retrans (sample) ==="
ss -ti "( dport = :${PORT} or sport = :${PORT} )" | sed -n '1,20p' || true
echo -e "\n=== mtr (${COUNT} cycles) ==="
command -v mtr >/dev/null && mtr -ezbw -c "${COUNT}" -T -P "${PORT}""${TARGET}" || echo"mtr not found"
echo -e "\n=== interface drops (ip -s link) ==="
ip -s link | sed -n '1,200p'
echo -e "\n=== NIC stats (ethtool -S) ==="
command -v ethtool >/dev/null && ethtool -S eth0 2>/dev/null | egrep -i 'drop|err|fifo|nobuf' || echo"ethtool or stats not available"
echo -e "\n=== stack counters (nstat) ==="
command -v nstat >/dev/null && nstat -az | egrep 'TcpRetransSegs|IpInDiscard|UdpInErrors' || echo"nstat not found"
echo -e "\n=== qdisc ==="
tc qdisc show dev eth0 || true
echo -e "\n=== PMTU / tracepath(5 hops) ==="
command -v tracepath >/dev/null && tracepath -n "${TARGET}" | sed -n '1,5p' || echo"tracepath not found"
echo -e "\n=== conntrack usage (if present) ==="
for f in /proc/sys/net/netfilter/nf_conntrack_{count,max}; do [[ -r "$f" ]] && echo"$f: $(cat $f)"; doneSample Output of Simulated Run:
=== curl timing (api.example.com) ===
code=200 conn=0.010s tls=0.031s ttfb=0.082s total=0.096s
=== TCP sockets RTT/retrans (sample) ===
ESTAB ... rtt:81.2/3.1 mss:1448 cwnd:10 retrans:2/47
=== interface drops ===
RX ... dropped 120 overrun 0
TX ... dropped 0 carrier 0
=== qdisc ===
qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap ...8. Change Action “Pocket Card” (Changes Requiring Approval, Providing Minimal Commands)
Recommendation: First test on one machine, collect metrics, then promote.
- • Switch Queue Scheduler:
sudo tc qdisc replace dev eth0 root fq_codelecho 'net.ipv4.tcp_congestion_control=bbr' | sudo tee -a /etc/sysctl.conf && sudo sysctl -psudo ethtool -G eth0 rx 2048 tx 1024echo 3 | sudo tee /sys/class/net/eth0/queues/rx-0/rps_cpussudo iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
# or equivalent rule for nft9. Review and Solidify: Turning “Occasional Incidents” into “Predictable Physical Quantities”
- • Add to monitoring:
rtt/ttfb,TcpRetransSegs/s,ip -s link dropped,conntrack usage,mtr visualization. - • Annotate MTU/Queue Algorithm/Congestion Control and Version when releasing changes.
- • Conduct a “Network Degradation Drill” quarterly (using
tc netemto inject latency and packet loss) to verify the robustness of application timeout/retry strategies.
Example (only in drill environment):
# Simulate 50ms latency and 1% packet loss
$ sudo tc qdisc add dev eth0 root netem delay 50ms loss 1%
# Restore
$ sudo tc qdisc del dev eth0 rootConclusion
The network is like an invisible river, with ebb and flow. Latency and packet loss are not mysticism; they have traces to follow and numbers to prove. Keep the “quick check” at your fingertips, engrave the “evidence” into the process, and make “fixes” into reversible changes. This way, when the next fog arises, you will see the riverbed’s texture earlier, rather than being startled by the sound of water.
Old Yang’s Time
Let me clarify first, in daily life, everyone calls me Bo Ge, which has nothing to do with seniority, mainly because I am older. It’s just a nickname.
My younger colleagues born in the 2000s I call all with Ge, Zhang Ge, Li Ge.
However, this title, when attending some offline events, is not very appropriate when the sponsors also call me that.
For example, last time a planning director from a certain group said at a company meeting: “Today we are happy! Please welcome Bo Ge from the IT operations and maintenance technical circle to say a few words.”
This atmosphere with this title in the internet industry is a bit mismatched!
Every time I encounter this situation, I want to respond: “Meeting you all is fate, thank you for your kindness, let’s not say anything, it’s all in the wine. I drink, you all are free!”
So from now on, let’s call me Old Yang, which is both down-to-earth and low-key. I think it’s quite nice.
Operations X Files series articles:
From Alarm to CTO: The 11-Hour Life-and-Death Race of a P0 Incident
Comprehensive Guide to Securing Enterprise-Level Kubernetes Clusters (Including One-Click Check Script)
Click the original link to access real-time VPS information query and small tool aggregation project
Old Yang’s AI Account