Background Information
Have you ever encountered a scenario where Service A calls Service B, but Service B crashes (or the network disconnects)? At this point, Service A’s request does not immediately return an error but instead gets stuck! It can be stuck for tens of seconds, or even a minute or two… 🤯This leads to a slowdown in the entire system response, potentially causing a cascade failure.Today, we will delve into one of the culprits behind this “foolish waiting”: the Linux kernel parameter net.ipv4.tcp_syn_retries.
Parameter Explanation
🧐 What is tcp_syn_retries?
In the TCP three-way handshake, the client sends the first packet SYN to the server to request a connection. If there is network congestion, the server crashes, or the firewall drops packets, and the client does not receive the server’s SYN+ACK response, what happens?It will retry!
net.ipv4.tcp_syn_retries is used to control the number of retries for the SYN packets.
The default value is usually 6 (CentOS 7) or 5 (some distributions).
⏱️ How long do we have to wait? (Hardcore Calculation)
Many people think that retrying 6 times means waiting 6 seconds, which is a big mistake! ❌
The TCP timeout retransmission mechanism follows the principle of Exponential Backoff. After the first SYN is sent, if there is no response:
Wait 1 second -> Retry 1st time
Wait 2 seconds -> Retry 2nd time
Wait 4 seconds -> Retry 3rd time
Wait 8 seconds -> Retry 4th time
Wait 16 seconds -> Retry 5th time
Wait 32 seconds -> Retry 6th time
Finally, wait 64 seconds to confirm failure -> Give up the connection
Total time ≈ 1 + 2 + 4 + 8 + 16 + 32 + 64 = 127 seconds! 😱
This means that if your Service B is completely unreachable, under the default configuration, Service A’s connection request will block for over 2 minutes before returning “Connection timed out”.
For a high-concurrency microservices architecture, this 2 minutes is enough to exhaust your thread pool, leading to service collapse.
Packet Capture Analysis
🔍 Packet capture evidence: Seeing is believing
Words are not enough; let’s capture a packet and take a look. Suppose we try to connect to a non-existent IP (or an IP dropped by the firewall), while using tcpdump to capture packets.
# Terminal 1: Capture packets
tcpdump -i eth0 dst port 80 -n
# Terminal 2: Initiate connection (assuming target IP 192.168.1.99 is unreachable)
curl 192.168.1.99
Packet capture result analysis:
21:00:00.000000 IP 10.0.0.5.45678 > 192.168.1.99.80: Flags [S], seq 1234567890, ...21:00:01.000000 IP 10.0.0.5.45678 > 192.168.1.99.80: Flags [S], seq 1234567890, ...21:00:03.000000 IP 10.0.0.5.45678 > 192.168.1.99.80: Flags [S], seq 1234567890, ...21:00:07.000000 IP 10.0.0.5.45678 > 192.168.1.99.80: Flags [S], seq 1234567890, ...👀 Key Point Interpretation:
Flags [S]: All packets are SYN packets, indicating that the handshake has not been successful.
Source port remains unchanged (45678): Note that 10.0.0.5.45678, the source port is always 45678. This proves that these packets belong to the same connection request, not new requests. If it were an application layer retry (e.g., curl retrying after a timeout), the source port would typically change.
Time intervals (Exponential Backoff):
1st retransmission: 1 second from the initial packet (21:00:01)
2nd retransmission: 2 seconds from the last one (21:00:03)
3rd retransmission: 4 seconds from the last one (21:00:07)
…and so on.
This is the kernel-level retransmission mechanism; the application layer (such as Nginx, Tomcat, Go HTTP client) is usually unaware of it until it receives the final error (unless the application layer sets a shorter timeout).
How to View and Modify
1. View current value
sysctl net.ipv4.tcp_syn_retries
# Or
cat /proc/sys/net/ipv4/tcp_syn_retries
2. Temporarily modify (takes effect immediately, lost on reboot)
sysctl -w net.ipv4.tcp_syn_retries=2
3. Permanent modification
Edit /etc/sysctl.conf, add or modify:
net.ipv4.tcp_syn_retries = 2
Then execute sysctl -p to take effect.
Best Practices
💡 Best Practice: What should it be set to?
This parameter does not have a standard answer; it depends on your network environment.
✅ Scenario 1: Internal microservices (Recommended value: 1 or 2)
Within a data center, the network is very stable with low latency (in milliseconds). If 1-2 retries (about 3-7 seconds) cannot connect, it can be determined that the other side is down or the network is disconnected. It is recommended to set it to 2.
Total time for 2 retries: 1 + 2 + 4 = 7 seconds.
Compared to 127 seconds, failing fast allows your application to attempt other nodes or quickly degrade.
✅ Scenario 2: Cross-public network/weak network environment (Recommended value: 4 or 5)
If it is a mobile connection to a server or a cross-country link, network jitter is common. In this case, if the number of retries is too low, a slight packet loss can cause the connection to drop, leading to a poor user experience. It is recommended to keep the default value or set it to 4.
⚠️ Notes
Do not set it to 0: Although theoretically possible, it is highly discouraged. If the network jitters slightly, all your connections will fail, which is too sensitive.
Distinguish tcp_retries2: tcp_syn_retries controls retries for establishing connections; while tcp_retries2 controls retries for data transmission after the connection is established. Do not confuse the two.
Summary
❶ tcp_syn_retries controls the number of retries for SYN handshake packets.
❷ The default value usually leads to a timeout wait of around 2 minutes.
❸ Internal services are recommended to be adjusted to 2 for fast failure.
❹ Public services should be cautious and retain enough retry opportunities.
⚠️ Check the configuration of this parameter in your environment now.If you find this article useful, feel free to like, share, and forward it! ❤️
Little Knowledge
The Linux screen command is used for managing multiple windows, with core functions as follows:
-
Multi-window management: Create multiple virtual windows in one terminal to run programs independently.
-
Session persistence: Tasks in the session can continue to run in the background even if the network disconnects or the terminal closes.
-
Reconnect: Disconnect from the session at any time and reconnect later to resume operations.
-
Session sharing: Allows multiple users to share the same session for collaboration.
-
Window splitting: Supports horizontal or vertical window splitting to view multiple task outputs simultaneously.
🙅 Stop using nohup & and try screen!
Today’s Overcast
Review Summary
Technical Progress Together
AI Operations Practice
<Previous: AI Crawler Practice: Using N8N to Write Public Account Articles into Notion with One Click