STM32 Gateway Performance Avalanche: Packet Loss and System Freeze with Multiple Connections, Resolved with LwIP and Custom Memory Pool

This is a project experience shared by a colleague of mine. They developed an industrial gateway based on STM32F767 for a factory, which is used for:

Collecting data from multiple Modbus/485 sensors
Uploading data to the dispatch server via TCP/UDP protocol
Supporting up to 32 devices connected simultaneously (LwIP + RAW API)

Initial small-scale tests went smoothly, but once it went live, the customer reported:

“As soon as more than 10 devices are connected, the gateway starts dropping connections and packets, and in severe cases, the entire system freezes, requiring a power cycle to restart.”

After testing:

CPU usage skyrocketed with the 8th connection
From the 12th connection, LwIP began to continuously report pbuf_alloc failed
From the 15th connection, the system fell into a loop of packet loss and crashed completely within seconds

🚨 Fault Phenomenon: Increasing Connection Count → Network Packet Loss → LwIP Anomalies → System Hang

LwIP logs continuously output:

mem_malloc: could not allocate memory
tcp_alloc: pbuf allocation failed
pbuf_alloc: Out of pbufs

Further manifestations:

TCP connections are constantly being reset
UDP packet reception failure rate > 80%
CPU usage is not high, but system response slows down or even hangs
Sometimes even the watchdog cannot recover

🔍 Preliminary Diagnosis: LwIP Default Memory Pool Cannot Handle High Concurrency, Severe Memory Fragmentation

The LwIP protocol stack uses multiple memory pools (mempools) to manage different objects:

Memory Pool	Usage
`<span>PBUF_POOL</span>`	Network packet buffer
`<span>TCP_SEG</span>`	TCP segment structure
`<span>NETCONN</span>`, `<span>RAW_PCB</span>`	Connection control block
`<span>MEM</span>`	Heap memory (e.g., `<span>mem_malloc()</span>`)

We enabled the following default configurations (generated by CubeMX):

#define MEM_SIZE       16000
#define MEMP_NUM_PBUF  16
#define PBUF_POOL_SIZE 16

This is sufficient for small data volumes, but when the number of connections reaches 10+, each connection occupies multiple pbuf, quickly leading to:

“pbuf pool exhausted, PBUF_ALLOC failed → LwIP cannot receive → packet loss → connection reset → crash”

We also found:

LwIP defaults to using mem_malloc() to allocate structures like TCP_SEG, leading to rapid memory fragmentation
Resource contention occurs during multi-task recv(), causing dynamic memory fragmentation in FreeRTOS

🧠 The Truth Revealed: LwIP’s Default Configuration Cannot Handle High Concurrency, Custom Memory Pools are Essential!

By analyzing LwIP mem_stats and runtime memory distribution, we confirmed:

Exhaustion of pbuf_pool is the root cause
mem_malloc() frequently allocates small objects, leading to severe fragmentation
MEMP_NUM_TCP_SEG is too small, causing TCP backlog and blocking transmission

✅ Solution: Three Steps to Build an STM32 Gateway Capable of Withstanding 10 Times the Connection Load

🧱 Step 1: Expand LwIP Memory Pools, Dedicated Pools for Each Object

We modified lwipopts.h:

#define MEM_SIZE             64000
#define MEMP_NUM_PBUF        128
#define MEMP_NUM_TCP_SEG     64
#define MEMP_NUM_NETCONN     32
#define PBUF_POOL_SIZE       128
#define PBUF_POOL_BUFSIZE    512

Reserving multiple pbufs + TCP SEG for each connection to avoid resource contention

Results:

Single connection usage is more elastic
No more occurrences of pbuf_alloc failed
Memory fragmentation significantly reduced

🧠 Step 2: Enable LwIP Custom Memory Pool Mechanism (memp_std) + Fixed-Length Allocation

We enabled:

#define MEM_USE_POOLS        1
#define MEMP_USE_CUSTOM_POOLS 1

And customized multiple memory block pools in lwippools.h:

LWIP_MEMPOOL_DECLARE(RXBUF, 64, 512, "RX Buffers")
LWIP_MEMPOOL_DECLARE(TXBUF, 64, 512, "TX Buffers")

Benefits:

Each type of data structure manages its own memory pool
Faster allocation speed (fixed-length blocks)
Avoids dynamic fragmentation from malloc()

🛡️ Step 3: Network Task Isolation + Receive Queue Rate Limiting to Prevent “Storm Attacks”

We implemented rate limiting for the network task structure:

Each connection can occupy a maximum of 8 pbufs
Using message queues to block recv(), preventing CPU idling
Automatically disconnecting abnormal connections (e.g., not reading data for a long time)

We also added a connection monitoring thread:

if (conn->recv_q_len > 8 || conn->idle_time > 10s) {
    netconn_close(conn);
    netconn_delete(conn);
}

To prevent individual devices from “dragging down the entire system”

🧪 Optimized Testing Results

We conducted a simulated “storm attack test”: connecting 60 TCP clients simultaneously, sending 1000 small packets per second.

Metric	Before Optimization	After Optimization
Maximum Connection Count	10~12	60+
Packet Loss Rate	> 75%	< 0.5%
LwIP Memory Overflow	Frequent	0 times
CPU Usage	High Fluctuation	Stable
System Stable Running Time	Minutes	Continuous 72 hours without crash
Customer Satisfaction	❌ Suspended Cooperation	✅ Resumed and Expanded Orders

🧰 Final Thoughts

While LwIP is lightweight, it is not a “plug-and-play” universal TCP/IP solution:

✅ High concurrency scenarios must redesign lwipopts.h
✅ Use MEMP custom pools instead of malloc() to avoid fragmentation
✅ Network tasks need rate limiting and isolation to prevent “global failure”
✅ Add monitoring threads to detect and clean up abnormal connections

Otherwise, you will face:“A disaster scene where too many connections lead to system avalanche!”