Mastering Linux Network Troubleshooting in One Article!

Network failures are one of the most common challenges faced by operations engineers. A single network outage can lead to tens of thousands of dollars in business losses, and the ability to quickly locate and resolve issues often determines the value of an operations engineer. This article shares my accumulated experience in network troubleshooting within an enterprise environment, helping you establish a systematic approach to fault handling.

🎯 The Golden Rules of Troubleshooting

Layered Troubleshooting Strategy

Network troubleshooting follows the OSI seven-layer model, analyzing from the physical layer to the application layer:

Physical Layer → Data Link Layer → Network Layer → Transport Layer → Application Layer

This bottom-up troubleshooting approach can quickly pinpoint the root cause of the problem, avoiding wasted time in the wrong direction.

🔧 Essential Toolbox

Basic Network Tools

# Connectivity Test
ping -c 4 target_IP
ping6 -c 4 target_IPv6

# Route Tracing
traceroute target_IP
mtr --report --report-cycles 10 target_IP

# Port Connectivity
telnet target_IP port
nc -zv target_IP port_range

Advanced Diagnostic Tools

# Network Traffic Analysis
tcpdump -i eth0 -w capture.pcap
wireshark   # Graphical Analysis

# Network Statistics
netstat -tulpn
ss -tulpn
lsof -i :port_number

# System Resource Monitoring
iotop    # IO Usage
iftop    # Real-time Network Traffic Monitoring

🚨 Common Fault Scenarios and Solutions

Scenario 1: Server Cannot Connect to the Internet

Fault Phenomenon:

  • • Internal network communication is normal
  • • Cannot ping external IP
  • • Domain name resolution fails

Troubleshooting Steps:

  1. 1. Check Local Network Configuration
# View IP Configuration
ip addr show
ip route show

# Check DNS Configuration
cat /etc/resolv.conf
nslookup google.com
  1. 2. Test Gateway Connectivity
# Get Default Gateway
ip route | grep default

# Test Gateway Connectivity
ping -c 4 gateway_IP
  1. 3. Check Firewall Rules
# CentOS/RHEL
firewall-cmd --list-all
iptables -L -n

# Ubuntu
ufw status

Solution:

  • • Configure the correct gateway and DNS
  • • Check firewall rules
  • • Verify routing table configuration

Scenario 2: Abnormal Network Latency

Fault Phenomenon:

  • • Connection timeout
  • • Slow response
  • • High packet loss rate

In-depth Analysis:

# Detailed ping test
ping -c 100 -i 0.1 target_IP

# Route hop analysis
mtr --report --report-cycles 100 target_IP

# Network quality test
iperf3 -c target_server

Performance Optimization:

# Adjust TCP parameters
echo 'net.core.rmem_max = 16777216' >> /etc/sysctl.conf
echo 'net.core.wmem_max = 16777216' >> /etc/sysctl.conf
sysctl -p

Scenario 3: Port Cannot Be Accessed

Fault Phenomenon:

  • • Service starts normally
  • • Port cannot be connected
  • • Firewall configuration is correct

Troubleshooting Process:

# Confirm service listening status
netstat -tlpn | grep :port_number
ss -tlpn | grep :port_number

# Check listening address
# 0.0.0.0 - listens on all interfaces
# 127.0.0.1 - listens only on local loopback

# Test local connection
telnet 127.0.0.1 port_number
curl -v http://127.0.0.1:port_number

Solution Strategy:

  1. 1. Modify service configuration to listen on the correct address
  2. 2. Check SELinux policies
  3. 3. Verify application configuration

📊 Practical Case Studies in Troubleshooting

Case 1: Database Connection Exception

Background: In the production environment, the application server suddenly cannot connect to the database

Troubleshooting Process:

# 1. Basic connectivity test
ping database_IP
telnet database_IP 3306

# 2. Check database service status
systemctl status mysql
netstat -tlpn | grep :3306

# 3. View error logs
tail -f /var/log/mysql/error.log

Problem Found: The number of connections to the database server has reached its limit

Solution:

# Temporary solution
mysql -u root -p -e "SHOW PROCESSLIST;"
mysql -u root -p -e "KILL connection_ID;"

# Permanent solution
vim /etc/mysql/mysql.conf.d/mysqld.cnf
max_connections = 1000

Case 2: Slow DNS Resolution

Problem Description: Website access speed is extremely slow, but direct access to the IP is normal

Analysis Process:

# DNS resolution time test
time nslookup domain.com

# Test different DNS servers
nslookup domain.com 8.8.8.8
nslookup domain.com 114.114.114.114

# Clear DNS cache
systemctl restart systemd-resolved

Optimization Plan:

# Configure faster DNS servers
echo "nameserver 8.8.8.8" > /etc/resolv.conf
echo "nameserver 114.114.114.114" >> /etc/resolv.conf

# Enable DNS caching
systemctl enable systemd-resolved

🛠️ Advanced Troubleshooting Techniques

Network Packet Analysis

# Capture packets on a specific port
tcpdump -i any -w debug.pcap port 80

# Analyze HTTP requests
tcpdump -i eth0 -A -s 1024 port 80

# Filter specific hosts
tcpdump -i eth0 host 192.168.1.100

Performance Bottleneck Identification

# Network interface statistics
cat /proc/net/dev
ip -s link show

# Connection status statistics
ss -s
netstat -s

Automated Monitoring Script

#!/bin/bash
# Network health check script
check_network() {
    local target=$1
    local port=$2
    
    # Connectivity check
    if ping -c 3 -W 2 $target >/dev/null; then
        echo "✅ $target connectivity normal"
    else
        echo "❌ $target connectivity abnormal"
        return 1
    fi
    
    # Port check
    if nc -z -w 3 $target $port >/dev/null; then
        echo "✅ $target:$port port normal"
    else
        echo "❌ $target:$port port abnormal"
        return 1
    fi
}

# Batch check
check_network "192.168.1.1" "22"
check_network "8.8.8.8" "53"

📈 Preventive Maintenance Strategies

Monitoring and Alert Configuration

# Use Zabbix to monitor network status
# Monitoring items:
# - Network interface traffic
# - Connection count statistics
# - Response time
# - Packet loss rate

# Set alert thresholds
# Latency > 100ms
# Packet loss rate > 1%
# Connection count > 80%

Daily Maintenance Checklist

  • • Health status of network devices
  • • Bandwidth usage
  • • Firewall log review
  • • DNS resolution performance
  • • Routing table integrity
  • • Network security scanning

🎓 Best Practices for Troubleshooting

1. Establish Standardized Processes

  • • Problem recording templates
  • • Troubleshooting step checklists
  • • Solution knowledge base

2. Tool Usage Techniques

  • • Proficient use of command-line tools
  • • Graphical tools for auxiliary analysis
  • • Automation scripts to improve efficiency

3. Continuous Learning

  • • Stay updated on new network technologies
  • • Participate in technical community exchanges
  • • Regularly review fault cases

💡 Summary

Network troubleshooting is a skill that requires a combination of theory and practice. Through systematic troubleshooting methods, appropriate tool usage, and rich practical experience, we can quickly locate and resolve various network issues.

Remember, every fault is an opportunity to learn. Build your own fault handling knowledge base and continuously improve your troubleshooting skills to maximize your value in critical moments.

Mastering Linux Network Troubleshooting in One Article!

Previous Recommendations

  • How has the architecture of large websites like “JD.com” evolved?

  • Why can’t your Nginx handle high concurrency?

  • Ansible, the powerful tool for batch server management, is very detailed!

  • Are we doing the CMDB model wrong in operations?

  • Keepalived, the artifact for dual-machine hot backup, achieves zero downtime!

  • Linux Shell tricks: sed

  • Cloud-based automation operations tool: Terraform

  • How to deal with mining viruses on online Linux servers?

  • Jenkins setup, permission management, parameterization, pipelines, etc., very detailed!

  • Detailed explanation of Jenkins Pipeline, recommended for collection!

  • Finally understood Nginx reverse proxy!

  • How to troubleshoot packet loss in Linux networks? Finally understood!

  • How does K8s achieve canary/gray releases?

  • Analysis of over 20 Linux commands for log analysis, very comprehensive!

  • Deploying a MySQL cluster on K8s, stable!

  • 40 common Nginx interview questions (with answers)

  • Jenkins production environment notes, very detailed!

  • 19 common issues with K8s clusters, recommended for collection

  • Nginx working principles and optimization summary (super detailed)

  • Six high-frequency Linux operation and maintenance troubleshooting notes!

  • 17 commonly used metrics in operations, 90% of people don’t know!

  • Why does performance drop so much after containerizing applications?

  • Detailed explanation of Nginx rate limiting, dealing with traffic surges and malicious attacks

  • How to troubleshoot when the Linux disk is full?

  • What to do when the CPU spikes to 900%?

  • Nginx performance optimization is covered in this article!

  • K8s Pod “OOM Killer”, found the reason

  • Summary of Nginx high concurrency performance optimization notes from large companies

  • Building a CI/CD system based on Jenkins

  • Seven major application scenarios of Nginx (with configuration)

  • The most comprehensive explanation of Jenkins Pipeline

  • Learning guide for the mainstream monitoring system Prometheus

  • 40 commonly asked Nginx interview questions

  • Common Linux operation and maintenance interview questions, a must-read for job seekers!

  • 25 common MySQL interview questions and answers

Leave a Comment