Linux Networking: Let the Page Fly [Part 1]

While troubleshooting an issue related to a network device, I finally found the root cause.

The phenomenon is that <span>dmesg</span> keeps printing the following log:

page_pool_release_retry() stalled pool shutdown 1 inflight 2899 sec

Root cause: The Kafka Go library IBM/sarama^[1] has a TCP connection leak issue (Client SeekBroker Connection Leak^[2]).

TL;DR: The leaked TCP connections lead to a causal chain where the page pool cannot reclaim pages:

sarama has a TCP connection leak
 → TCP sockets remain in TCP_CLOSE state
   → sockets sk_receive_queues cache FINACK skb
     → those skb reference pages allocated from the page pool
       → page_pool_release_retry() fails to reclaim pages
         → page_pool_release_retry() keeps trying to reclaim, continuously failing
           → logs printed to dmesg

1. Reproducing the Issue

The machine where the issue was discovered uses a Mellanox network card; after repeated attempts, it was found that simply executing <span>ip link set dev eth0 down</span> and waiting for 1 minute would show logs in <span>dmesg</span>.

1.1. What Happens When the Network Device is Turned Off?

1. The network driver returns pages to the page pool when destroying the queue:<span>page_pool_put_defragged_page()</span>.
2. Releases the page pool:<span>page_pool_destroy()</span>.
3. If there are inflight pages, a worker is started to periodically (every second) attempt to reclaim pages and release the page pool via <span>page_pool_release_retry()</span>.

1.2. Why Wait 1 Minute?

Directly checking the source code of <span>page_pool_release_retry()</span>:

// net/core/page_pool.c

#define DEFER_WARN_INTERVAL (60 * HZ)

static void page_pool_release_retry(struct work_struct *wq)
{
    struct delayed_work *dwq = to_delayed_work(wq);
    struct page_pool *pool = container_of(dwq, typeof(*pool), release_dw);
    int inflight;

    inflight = page_pool_release(pool);
    /* In rare cases, a driver bug may cause inflight to go negative.
     * Don't reschedule release if inflight is 0 or negative.
     * - If 0, the page_pool has been destroyed
     * - if negative, we will never recover
     * in both cases no reschedule is necessary.
     */
    if (inflight <= 0)
        return;

    /* Periodic warning */
    if (time_after_eq(jiffies, pool->defer_warn)) {
        int sec = (s32)((u32)jiffies - (u32)pool->defer_start) / HZ;

        pr_warn("%s() stalled pool shutdown %d inflight %d sec\n",
            __func__, inflight, sec);
        pool->defer_warn = jiffies + DEFER_WARN_INTERVAL;
    }

    /* Still not ready to be disconnected, retry later */
    schedule_delayed_work(&pool->release_dw, DEFER_TIME);
}

Only when it exceeds <span>pool->defer_warn</span> (1 minute) will it print logs; thus, when <span>page_pool_release_retry()</span> runs for the first time, it will not print logs.

Moreover:

1. In the first minute, it is likely to reclaim many pages; therefore, printing logs during this time is meaningless and will create noise.
2. The optimization added by Jason Xing <span>if (inflight <= 0)</span> does not work because <span>inflight</span> is greater than 0, indicating that there are indeed pages that have not been returned.

1.3. After Waiting 1 Minute, There May Be No Logs in dmesg

Before uncovering more details, it was unclear whether there were still pages that had not been returned.

1.4. Turning Point in Troubleshooting

Even consulting AI was often futile; AI can provide ideas and directions, but its effectiveness is limited.

In <span>page_pool.c</span>, there are 2 tracepoints that can be used to track the allocation and reclamation of pages:

# bpfsnoop --show-func-proto -t 'page_pool_state_*'
Kernel tracepoints: (total 2)
void page_pool_state_hold(const struct page_pool *pool, const struct page *page, u32 hold);
void page_pool_state_release(const struct page_pool *pool, const struct page *page, u32 release);

However, even if these 2 tracepoints are used to analyze which pages have not been reclaimed, it is still impossible to know the reason for those pages not being reclaimed.

[Progress] Continuously observing dmesg, it was found that after a period of time, those logs would decrease or even stop printing.

In other words: during this process, some pages were returned and reclaimed.

The next step is to deploy bpfsnoop:

# bpfsnoop \
    -k '(s)page_pool_put_*' \
    -t '(s)page_pool_state_*' \
        --filter-arg 'pool->destroy_cnt > 1' \
        --output-arg 'pool->destroy_cnt' \
        --output-arg 'pool->pages_state_hold_cnt' \
        --output-arg 'pool->pages_state_release_cnt.counter' \
        --output-arg '*page' -o inflight_page.bpfsnoop.log

← page_pool_put_defragged_page args=((struct page_pool *)pool=0xff110001fde50800, (struct page *)page=0xffd4000007fc18c0, (unsigned int)dma_sync_size=0xffffffff/4294967295, (bool)allow_direct=false) retval=(void) cpu=22 process=(262275:xxx-agent) timestamp=17:39:16.551295117
Arg attrs: (u64)'pool->destroy_cnt'=0x45/69, (u32)'pool->pages_state_hold_cnt'=0x1000/4096, (int)'pool->pages_state_release_cnt.counter'=4095, (struct page)'*page'={"flags": 6755398367313920, "": {"": {"": {"lru": {"next": 0xdead000000000040, "prev": 0xff110001fde50800}, "": {"__filler": 0xdead000000000040, "mlock_count": 4259645440}, "buddy_list": {"next": 0xdead000000000040, "prev": 0xff110001fde50800}, "pcp_list": {"next": 0xdead000000000040, "prev": 0xff110001fde50800}}, "mapping": 0x0, "": {"index": 8573562880, "share": 8573562880}, "private": 1}, "": {"pp_magic": 16045481047390945344, "pp": 0xff110001fde50800, "_pp_mapping_pad": 0, "dma_addr": 8573562880, "": {"dma_addr_upper": 1, "pp_frag_count": {"counter": 1}}}, "": {"compound_head": 16045481047390945344}, "": {"pgmap": 0xdead000000000040, "zone_device_data": 0xff110001fde50800}, "callback_head": {"next": 0xdead000000000040, "func": 0xff110001fde50800}}, "": {"_mapcount": {"counter": -1}, "page_type": 4294967295}, "_refcount": {"counter": 1}, "memcg_data": 0}
Func stack:
  page_pool_put_defragged_page+0x5                      ; net/core/page_pool.c:640
  skb_free_head+0x55                                    ; net/core/skbuff.c:953
  skb_release_data+0x159                                ; net/core/skbuff.c:998
  __kfree_skb+0x2b                                      ; net/core/skbuff.c:1068
  __tcp_close+0x93                                      ; include/linux/skbuff.h:2065
  inet_release+0x44                                     ; net/ipv4/af_inet.c:435
  __sock_release+0x40                                   ; net/socket.c:660
  sock_close+0x15                                       ; net/socket.c:1423
  __fput+0xfe                                           ; fs/file_table.c:385
  ____fput+0xe                                          ; fs/file_table.c:413
  task_work_run+0x65                                    ; arch/x86/include/asm/jump_label.h:27
  do_exit+0x298                                         ; kernel/exit.c:884
  do_group_exit+0x35                                    ; kernel/exit.c:1006
  get_signal+0x95f                                      ; kernel/signal.c:2902
  arch_do_signal_or_restart+0x39
  exit_to_user_mode_loop+0x9a                           ; kernel/entry/common.c:176
  exit_to_user_mode_prepare+0xa5                        ; kernel/entry/common.c:210
  syscall_exit_to_user_mode+0x29
  do_syscall_64+0x62                                    ; arch/x86/entry/common.c:88
  entry_SYSCALL_64_after_hwframe+0x78                   ; kernel/entry/entry_64.S:121

After some time, analyzing the bpfsnoop log file revealed the above call stack: some pages were released through <span>__kfree_skb()</span><code><span> when the socket was released.</span>

2. Protocol Stack Behavior Analysis

From the above call stack, it involves TCP sockets, especially during the TCP socket release phase.

2.1. Why Does <span>__tcp_close()</span><code><span> Release skb?</span>

Directly checking the source code of <span>__tcp_close()</span><code><span> to see why it calls </span><code><span>__kfree_skb()</span><code><span>.</span>

// net/ipv4/tcp.c

void __tcp_close(struct sock *sk, long timeout)
{
    struct sk_buff *skb;
    int data_was_unread = 0;
    int state;

    WRITE_ONCE(sk->sk_shutdown, SHUTDOWN_MASK);

    if (sk->sk_state == TCP_LISTEN) {
        tcp_set_state(sk, TCP_CLOSE);

        /* Special case. */
        inet_csk_listen_stop(sk);

        goto adjudge_to_death;
    }

    /*  We need to flush the recv. buffs.  We do this only on the
     *  descriptor close, not protocol-sourced closes, because the
     *  reader process may not have drained the data yet!
     */
    while ((skb = __skb_dequeue(&sk->sk_receive_queue)) != NULL) {
        u32 len = TCP_SKB_CB(skb)->end_seq - TCP_SKB_CB(skb)->seq;

        if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
            len--;
        data_was_unread += len;
        __kfree_skb(skb);
    }

    /* If socket has been already reset (e.g. in tcp_reset()) - kill it. */
    if (sk->sk_state == TCP_CLOSE)
        goto adjudge_to_death;

    /*...*/

out:
    bh_unlock_sock(sk);
    local_bh_enable();
}

It turns out that it is clearing <span>sk->sk_receive_queue</span>.

2.2. How Does <span>__kfree_skb()</span><code><span> Reclaim Pages?</span>

The information provided by the above call stack does not explain why <span>__kfree_skb()</span><code><span> reclaims pages.</span>

Directly digging into the source code of <span>__kfree_skb()</span><code><span>:</span>

// net/core/skb.c

/*
__kfree_skb()
|-->skb_release_all()
    |-->skb_release_data()
        |-->skb_free_head()
            |-->skb_pp_recycle()
                |-->napi_pp_put_page()
                    |-->page_pool_put_full_page()
*/

void __kfree_skb(struct sk_buff *skb)
{
    skb_release_all(skb, SKB_DROP_REASON_NOT_SPECIFIED, false);
    kfree_skbmem(skb);
}
EXPORT_SYMBOL(__kfree_skb);

/* Free everything but the sk_buff shell. */
static void skb_release_all(struct sk_buff *skb, enum skb_drop_reason reason,
                bool napi_safe)
{
    skb_release_head_state(skb);
    if (likely(skb->head))
        skb_release_data(skb, reason, napi_safe);
}

static void skb_release_data(struct sk_buff *skb, enum skb_drop_reason reason,
                 bool napi_safe)
{
    struct skb_shared_info *shinfo = skb_shinfo(skb);
    int i;

    if (skb->cloned &&
        atomic_sub_return(skb->nohdr ? (1 << SKB_DATAREF_SHIFT) + 1 : 1,
                  &shinfo->dataref))
        gotoexit;

    if (skb_zcopy(skb)) {
        bool skip_unref = shinfo->flags & SKBFL_MANAGED_FRAG_REFS;

        skb_zcopy_clear(skb, true);
        if (skip_unref)
            goto free_head;
    }

    for (i = 0; i < shinfo->nr_frags; i++)
        napi_frag_unref(&shinfo->frags[i], skb->pp_recycle, napi_safe);

free_head:
    if (shinfo->frag_list)
        kfree_skb_list_reason(shinfo->frag_list, reason);

    skb_free_head(skb, napi_safe);
exit:
    /* When we clone an SKB we copy the recycling bit. The pp_recycle
     * bit is only set on the head though, so in order to avoid races
     * while trying to recycle fragments on __skb_frag_unref() we need
     * to make one SKB responsible for triggering the recycle path.
     * So disable the recycling bit if an SKB is cloned and we have
     * additional references to the fragmented part of the SKB.
     * Eventually the last SKB will have the recycling bit set and it's
     * dataref set to 0, which will trigger the recycling
     */
    skb->pp_recycle = 0;
}

static void skb_free_head(struct sk_buff *skb, bool napi_safe)
{
    unsignedchar *head = skb->head;

    if (skb->head_frag) {
        if (skb_pp_recycle(skb, head, napi_safe))
            return;
        skb_free_frag(head);
    } else {
        skb_kfree_head(head, skb_end_offset(skb));
    }
}

static bool skb_pp_recycle(struct sk_buff *skb, void *data, bool napi_safe)
{
    if (!IS_ENABLED(CONFIG_PAGE_POOL) || !skb->pp_recycle)
        returnfalse;
    return napi_pp_put_page(virt_to_page(data), napi_safe);
}

bool napi_pp_put_page(struct page *page, bool napi_safe)
{
    bool allow_direct = false;
    struct page_pool *pp;

    page = compound_head(page);

    /* page->pp_magic is OR'ed with PP_SIGNATURE after the allocation
     * in order to preserve any existing bits, such as bit 0 for the
     * head page of compound page and bit 1 for pfmemalloc page, so
     * mask those bits for freeing side when doing below checking,
     * and page_is_pfmemalloc() is checked in __page_pool_put_page()
     * to avoid recycling the pfmemalloc page.
     */
    if (unlikely((page->pp_magic & ~0x3UL) != PP_SIGNATURE))
        returnfalse;

    pp = page->pp;

    /* Allow direct recycle if we have reasons to believe that we are
     * in the same context as the consumer would run, so there's
     * no possible race.
     * __page_pool_put_page() makes sure we're not in hardirq context
     * and interrupts are enabled prior to accessing the cache.
     */
    if (napi_safe || in_softirq()) {
        conststruct napi_struct *napi = READ_ONCE(pp->p.napi);

        allow_direct = napi &&
            READ_ONCE(napi->list_owner) == smp_processor_id();
    }

    /* Driver set this to memory recycling info. Reset it on recycle.
     * This will *not* work for NIC using a split-page memory model.
     * The page will be returned to the pool here regardless of the
     * 'flipped' fragment being in use or not.
     */
    page_pool_put_full_page(pp, page, allow_direct);

    returntrue;
}
EXPORT_SYMBOL(napi_pp_put_page);

Key points:

1. <span>skb->head_frag</span>
2. <span>skb->pp_recycle</span>

Is there a way to traverse TCP sockets and check the <span>sk->sk_receive_queue</span> for skb?

2.3. Trying BPF’s <span>iter/tcp</span>

Generated BPF code with AI, adjusted as follows:

//go:build ignore
// SPDX-License-Identifier: GPL-2.0
/* Copyright 2025 Leon Hwang */

#include "bpf_all.h"

SEC("iter/tcp")
int iter_tcp(struct bpf_iter__tcp *ctx)
{
    struct seq_file *seq = ctx->meta->seq;
    struct sock_common *skc = ctx->sk_common;
    struct tcp_sock *tp;
    struct sock *sk;
    struct sk_buff_head *queue;
    struct sk_buff *skb;
    int qlen, cnt = 0, cnt_pp = 0;

    if (!skc)
        return0;

    tp = bpf_skc_to_tcp_sock(skc);
    if (!tp)
        return0;

    sk   = (struct sock *)tp;
    queue = &sk->sk_receive_queue;

    qlen = BPF_CORE_READ(queue, qlen);

    skb = (struct sk_buff *)BPF_CORE_READ(queue, next);
    for (int i = 0; i < 100 && skb != (struct sk_buff *)queue; i++) {
        cnt++;
        cnt_pp += BPF_CORE_READ_BITFIELD_PROBED(skb, pp_recycle);

        skb = (struct sk_buff *)BPF_CORE_READ(skb, next);
    }

        if (cnt == 0)
                return0;

    BPF_SEQ_PRINTF(seq, "state=%d src=%pI4:%u dst=%pI4:%u\n",
               skc->skc_state,
               &skc->skc_rcv_saddr, skc->skc_num,
               &skc->skc_daddr, bpf_ntohs(skc->skc_dport));

    BPF_SEQ_PRINTF(seq, "  rx_queue: qlen=%d iterated=%d (pp_recycle=%d)\n",
               qlen, cnt, cnt_pp);

    return0;
}

However, when running, nothing appeared.

Need to think of another way.

2.4. Using Kernel Module to Traverse TCP Sockets

Using a kernel module to traverse all TCP sockets for a specified PID is definitely reliable and can obtain all necessary information.

#include <linux/module.h>
#include <linux/pid.h>
#include <linux/sched.h>
#include <linux/fs.h>
#include <linux/file.h>
#include <linux/fdtable.h>
#include <net/sock.h>
#include <net/inet_sock.h>
#include <net/tcp.h>
#include <net/tcp_states.h>
#include <linux/ip.h>
#include <linux/tcp.h>
#include <linux/inet.h>
#include <net/ipv6.h>
#include <net/page_pool/helpers.h>

staticpid_t target_pid;

static const char *tcp_state_str(int state)
{
    switch (state) {
    case TCP_ESTABLISHED: return"ESTABLISHED";
    case TCP_SYN_SENT:    return"SYN_SENT";
    case TCP_SYN_RECV:    return"SYN_RECV";
    case TCP_FIN_WAIT1:   return"FIN_WAIT1";
    case TCP_FIN_WAIT2:   return"FIN_WAIT2";
    case TCP_TIME_WAIT:   return"TIME_WAIT";
    case TCP_CLOSE:       return"CLOSE";
    case TCP_CLOSE_WAIT:  return"CLOSE_WAIT";
    case TCP_LAST_ACK:    return"LAST_ACK";
    case TCP_LISTEN:      return"LISTEN";
    case TCP_CLOSING:     return"CLOSING";
    case TCP_NEW_SYN_RECV: return"NEW_SYN_RECV";
    default:              return"UNKNOWN";
    }
}

static void log_pp_recycled_skb(struct sock *sk, unsigned long ino,
                unsigned int qlen, struct sk_buff *skb)
{
    struct inet_sock *inet = inet_sk(sk);
    struct tcp_sock *tp = tcp_sk(sk);
    struct skb_shared_info *shinfo = skb_shinfo(skb);
    char src_ip[64] = {0};
    char dst_ip[64] = {0};
    u16 src_port = 0, dst_port = 0;
    unsignedint i;

    if (sk->sk_family == AF_INET) {
        snprintf(src_ip, sizeof(src_ip), "%pI4", &inet->inet_saddr);
        snprintf(dst_ip, sizeof(dst_ip), "%pI4", &inet->inet_daddr);
        src_port = ntohs(inet->inet_sport);
        dst_port = ntohs(inet->inet_dport);
#if IS_ENABLED(CONFIG_IPV6)
    } elseif (sk->sk_family == AF_INET6) {
        snprintf(src_ip, sizeof(src_ip), "%pI6c", &sk->sk_v6_rcv_saddr);
        snprintf(dst_ip, sizeof(dst_ip), "%pI6c", &sk->sk_v6_daddr);
        src_port = ntohs(inet->inet_sport);
        dst_port = ntohs(inet->inet_dport);
#endif
    }

    pr_info("pid %d socket 0x%llx ino=%lu state=%u/%s shutdown=0x%x err=%d(soft=%d) linger2=%d dead=%u: receive_queue(qlen=%u): skb 0x%llx pp_recycle=%d tcp_flags=0x%02x cloned=%u head_frag=%u %s:%u->%s:%u len=%u data_len=%u nr_frags=%u\n",
        target_pid, (unsignedlonglong)sk, ino, sk->sk_state, tcp_state_str(sk->sk_state),
        sk->sk_shutdown, sk->sk_err, sk->sk_err_soft, tp->linger2, sock_flag(sk, SOCK_DEAD),
        qlen, (unsignedlonglong)skb, skb->pp_recycle,
        TCP_SKB_CB(skb)->tcp_flags,
        skb->cloned, skb->head_frag,
        src_ip, src_port, dst_ip, dst_port, skb->len, skb->data_len, shinfo->nr_frags);

    /* Log linear buffer head page if present */
    if (skb->head) {
        struct page *head_page = virt_to_head_page(skb->head);
        dma_addr_t head_dma = page_pool_get_dma_addr(head_page);
        pr_info("  pid %d head: page=0x%llx pfn=%lu data_off=%u headlen=%u dma=0x%llx\n",
            target_pid, (unsignedlonglong)head_page, page_to_pfn(head_page),
            (unsignedint)(skb->data - skb->head), skb_headlen(skb),
            (unsignedlonglong)head_dma);
    }

    for (i = 0; i < shinfo->nr_frags; i++) {
        constskb_frag_t *frag = &shinfo->frags[i];
        struct page *page = skb_frag_page(frag);
        dma_addr_t dma = page_pool_get_dma_addr(page);

        pr_info("  pid %d frag[%u]: page=0x%llx pfn=%lu off=%u size=%u dma=0x%llx\n",
            target_pid, i, (unsignedlonglong)page, page_to_pfn(page),
            skb_frag_off(frag), skb_frag_size(frag),
            (unsignedlonglong)dma);
    }
}

static void inspect_socket(struct socket *sock, unsigned long ino)
{
    struct sock *sk = sock->sk;
    struct sk_buff *skb;
    unsignedint qlen;
    int owned_before;

    if (!sk || sock->type != SOCK_STREAM || sk->sk_protocol != IPPROTO_TCP || sk->sk_state != TCP_CLOSE)
        return;

    owned_before = sock_owned_by_user(sk);
    lock_sock(sk);

    /* Log if we had to wait for lock */
    if (owned_before)
        pr_info("pid %d socket %p ino=%lu: WAITED for lock (was owned=%d)\n",
            target_pid, sk, ino, owned_before);

    qlen = skb_queue_len(&sk->sk_receive_queue);

    /* Check if there's backlog that will be processed */
    if (sk->sk_backlog.tail) {
        pr_info("pid %d socket %p ino=%lu: HAS BACKLOG (tail=0x%llx)\n",
            target_pid, sk, ino, (unsignedlonglong)sk->sk_backlog.tail);
    }

    skb_queue_walk(&sk->sk_receive_queue, skb) {
        log_pp_recycled_skb(sk, ino, qlen, skb);
    }
    release_sock(sk);
}

static bool inspect_fd_entry(struct files_struct *files, unsigned int fd)
{
    struct fdtable *fdt;
    struct file __rcu **fdentry;
    struct file *file;
    struct socket *sock;
    unsignedlong ino = 0;

    rcu_read_lock();
    fdt = files_fdtable(files);
    if (fd >= fdt->max_fds) {
        rcu_read_unlock();
        returnfalse;
    }

    fdentry = fdt->fd + array_index_nospec(fd, fdt->max_fds);
    file = rcu_dereference_raw(*fdentry);
    if (!file || !get_file_rcu(file)) {
        rcu_read_unlock();
        returntrue;
    }
    if (unlikely(files_fdtable(files) != fdt) ||
        unlikely(rcu_dereference_raw(*fdentry) != file)) {
        fput(file);
        rcu_read_unlock();
        returntrue;
    }
    rcu_read_unlock();

    if (file_inode(file))
        ino = file_inode(file)->i_ino;

    sock = sock_from_file(file);
    if (sock)
        inspect_socket(sock, ino);
    fput(file);

    returntrue;
}

static void walk_task_sockets(struct files_struct *files)
{
    unsignedint fd;

    for (fd = 0; inspect_fd_entry(files, fd); fd++)
        ;
}

static int __init qinspect_init(void)
{
    struct pid *pid;
    struct task_struct *task;
    struct files_struct *files = NULL;
    int ret = 0;

    pid = find_get_pid(target_pid);
    if (!pid)
        return -ESRCH;

    task = get_pid_task(pid, PIDTYPE_PID);
    put_pid(pid);
    if (!task)
        return -ESRCH;

    task_lock(task);
    files = task->files;
    if (files)
        atomic_inc(&files->count);
    task_unlock(task);

    if (!files) {
        ret = -ENOENT;
        goto out_put_task;
    }

    walk_task_sockets(files);

    atomic_dec(&files->count);

out_put_task:
    put_task_struct(task);
    return ret;
}

static void __exit qinspect_exit(void) { }

module_param(target_pid, int, 0444);
module_init(qinspect_init);
module_exit(qinspect_exit);
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Leon Hwang &lt;[email protected]&gt;");
MODULE_DESCRIPTION("TCP Socket sk_receive_queue Inspection Module");

Makefile required:

KDIR ?= /lib/modules/$(shell uname -r)/build

KMOD_NAME := tcp_rxq
KMOD := $(KMOD_NAME).ko

obj-m := $(KMOD_NAME).o
$(KMOD_NAME)-objs := $(KMOD_NAME).kmod.o

.PHONY: all clean

all:
    $(MAKE) -C $(KDIR) M=$(CURDIR) modules

clean:
    $(MAKE) -C $(KDIR) M=$(CURDIR) clean

run:
    if [ -z $(PID) ]; then exit 1; fi
    insmod $(KMOD) target_pid=$(PID)
    rmmod $(KMOD_NAME)
    dmesg | grep $(PID)

Usage:<span>make run PID=$(pidof xxx-agent)</span>.

After running, you can see:

[102694.540568] pid 286275 socket 0xff110002e0c53600 ino=4741242 state=7/CLOSE shutdown=0x3 err=32(soft=0) linger2=0 dead=0: receive_queue(qlen=1): skb 0xff110002b6aa6100 pp_recycle=1 tcp_flags=0x11 cloned=0 head_frag=1 10.x.x.x:43542->10.y.y.y:9093 len=0 data_len=0 nr_frags=0
[102694.540576]   pid 286275 head: page=0xffd400000f7bd040 pfn=4058945 data_off=118 headlen=0 dma=0x3def41000

Key information:

1. Many sockets are in the state <span>TCP_CLOSE</span><span>.</span>
2. <span>sk->sk_receive_queue</span><span> caches a </span><strong><span>FINACK</span></strong><span> skb (</span><code><span>tcp_flags=0x11</span><span>).</span>
3. The destination port is 9093.
4. <span>pp_recycle=1</span>.
5. This skb references a page.

So far, the behavior of printing logs in dmesg can be explained.

However, the matter is not over:

1. Why did the states of those sockets change to <span>TCP_CLOSE</span><span>?</span>
2. What is the root cause?
3. How to fundamentally solve the problem?
4. What areas can be improved?

References[1]

IBM/sarama: https://github.com/IBM/sarama

[2]

Client SeekBroker Connection Leak: https://github.com/IBM/sarama/issues/3143