Introduction
This article is recorded in February 2025 and is documented here.
In the production environment, one of the business system servers has had its memory consistently around 95% for the past few months. Last year, this system was reported as an innovative system, so preparations for the innovative components began right after the New Year, in conjunction with the acceptance of the innovative system. After the business system went live in November last year, I did not pay much attention to the server’s memory issue, as I thought their services required that much memory. It wasn’t until a few days ago during a routine inspection that I discovered a microservice had shut down abnormally, which was strange. Upon investigation, I found that it had triggered the system <span>OOM</span> mechanism, so I conducted some checks:
1. Check the top 10 processes by memory usage to see if there are any non-application system processes consuming a large amount of memory.
2. Based on the memory growth over the past few days (one month, three months), determine if a memory leak has occurred.
Background
Application Server: Kylin
[root@kylinV10arm162 rpm]# nkvers
############## Kylin Linux Version #################
Release:
Kylin Linux Advanced Server release V10 (Sword)
Kernel:
4.19.90-24.4.v2101.ky10.aarch64
Build:
Kylin Linux Advanced Server
release V10 (SP2) /(Sword)-aarch64-Build09/20210524
#################################################CPU: Kunpeng
[root@kylinV10arm162 rpm]# lscpu
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Thread(s) per core: 1
Core(s) per socket: 16
Socket(s): 1
NUMA node(s): 1
Vendor ID: HiSilicon
Model: 0
Model name: Kunpeng-920 # Look here **************
Stepping: 0x1
BogoMIPS: 200.00
NUMA node0 CPU(s): 0-15
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Not affected
Vulnerability Spectre v1: Mitigation; __user pointer sanitization
Vulnerability Spectre v2: Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp asimdfhm
The high memory usage has persisted, with memory usage increasing from 28G to 64G over two months, leading to the OOM mechanism being triggered multiple times, causing the application service to be killed.
Investigation
Checking the top 20 processes by memory usage, it can be seen that the <span>/sbin/auditd</span> process occupies <span>43.4%</span> of the memory.
[root@kylinV10arm162 log]# ps aux --sort=-%mem | head -n 20
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 925 0.0 43.4 42441024 28531008 ? S<sl 2024 44:19 /sbin/auditd
root 2897105 4.0 13.7 19020928 9049664 ? Sl 2024 2273:54 java -Dunisql.dll.dir= -Dloader.path=./lib -Xmx8g -Xms8g -XX:+HeapDumpOnOutOfMemoryError -DAsyncLogger.RingBufferSize=262144 -DAsyncLoggerConfig.RingBufferSize=262144 -Dlog4j2.AsyncQueueFullPolicy=Discard -DAppPID -jar hsrcm-rcce-task-202401.09.00-small.jar jfile=config/application.properties
root 824742 0.9 6.0 14404480 3962944 ? Sl 2024 1037:23 java -Dunisql.dll.dir= -jar -DAppPID -Dloader.path=./lib -Xms3g -Xmx3g -XX:+HeapDumpOnOutOfMemoryError -DAsyncLogger.RingBufferSize=262144 -DAsyncLoggerConfig.RingBufferSize=262144 -Dlog4j2.AsyncQueueFullPolicy=Discard hsrcm-amlps-server-1.55.0-small.jar jfile=config/application.properties
root 2883349 0.4 3.7 12539456 2450880 ? Sl 2024 244:16 java -Dunisql.dll.dir= -Dloader.path=./lib -Xmx2g -Xms2g -XX:+HeapDumpOnOutOfMemoryError -DAsyncLogger.RingBufferSize=262144 -DAsyncLoggerConfig.RingBufferSize=262144 -Dlog4j2.AsyncQueueFullPolicy=Discard -DAppPID -jar hsrcm-pbs-task-202401.09.02-small.jar jfile=config/application.properties
root 3589756 3.3 2.9 14083392 1962112 ? Sl Jan16 964:28 java -Dunisql.dll.dir= -jar -DAppPID -Dloader.path=./lib -Xms3g -Xmx3g -XX:+HeapDumpOnOutOfMemoryError -DAsyncLogger.RingBufferSize=262144 -DAsyncLoggerConfig.RingBufferSize=262144 -Dlog4j2.AsyncLogger.RingBufferSize=262144 -Dlog4j2.AsyncQueueFullPolicy=Discard hsrcm-amlcom-server-1.55.0-small.jar jfile=config/application.properties
root 1189277 14.8 2.4 12900544 1598592 ? Sl 2024 13005:10 java -Dloader.path=./lib -DDUBBO_IP_TO_REGISTRY=10.6.7.162 -Xms2g -Xmx2g -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/wf-server//logs/heapdump_20241206122204.hprof -XX:+PrintGCDetails -XX:+PrintClassHistogram -DAsyncLogger.RingBufferSize=262144 -DAsyncLoggerConfig.RingBufferSize=262144 -Dlog4j2.AsyncQueueFullPolicy=Discard -Xbootclasspath/a:config/ -jar -DAppPID wf-server.jar
root 2882269 1.5 2.2 11813120 1486144 ? Sl 2024 844:46 java -Dunisql.dll.dir= -Dloader.path=./lib -Xmx1g -Xms1g -XX:+HeapDumpOnOutOfMemoryError -DAsyncLogger.RingBufferSize=262144 -DAsyncLoggerConfig.RingBufferSize=262144 -Dlog4j2.AsyncQueueFullPolicy=Discard -jar hsrcm-pbs-server-202401.09.00-small.jar jfile=config/application.properties
root 454836 0.6 1.9 12947520 1301696 ? Sl 2024 596:04 java -Dunisql.dll.dir= -jar -DAppPID -Dloader.path=./lib -Xms2g -Xmx2g -XX:+HeapDumpOnOutOfMemoryError -DAsyncLogger.RingBufferSize=262144 -DAsyncLoggerConfig.RingBufferSize=262144 -Dlog4j2.AsyncQueueFullPolicy=Discard hsrcm-amlwl-server-1.55.0-small.jar jfile=config/application.properties
root 497527 0.4 1.9 12268160 1252608 ? Sl 2024 1350:55 java -Xms2048m -Xmx2048m -XX:MetaspaceSize=128m -Djava.net.preferIPv4Stack=true -classpath /home/jresconsole/jrescloud/jresconsole.mgr/jresconsole.mgr/conf:/home/jresconsole/jrescloud/jresconsole.mgr/jresconsole.mgr/lib/* com.hundsun.service_government.MgrApplication
root 3310093 0.8 1.8 12669568 1198016 ? Sl Jan15 263:14 java -Dloader.path=/home/bizframe/lib -jar -DAppPID -DAsyncLogger.RingBufferSize=262144 -DAsyncLoggerConfig.RingBufferSize=262144 -Dlog4j2.AsyncQueueFullPolicy=Discard -Dlog4j2.DiscardThreshold=ERROR -Xmx2G -Xms1G -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8 -DDUBBO_IP_TO_REGISTRY=10.6.7.162 bizframe-cloud-server-2.1.33.jar
root 834678 0.6 1.7 12820032 1174208 ? Sl 2024 674:30 java -Dunisql.dll.dir= -jar -DAppPID -Dloader.path=./lib -Xms2g -Xmx2g -XX:+HeapDumpOnOutOfMemoryError -DAsyncLogger.RingBufferSize=262144 -DAsyncLoggerConfig.RingBufferSize=262144 -Dlog4j2.AsyncQueueFullPolicy=Discard hsrcm-amlcrs-server-1.55.0-small.jar jfile=config/application.properties
root 671464 0.4 1.3 11144896 866112 ? Sl 2024 554:56 java -Dunisql.dll.dir= -Dloader.path=./lib -Xmx512m -Xms512m -XX:+HeapDumpOnOutOfMemoryError -DAsyncLogger.RingBufferSize=262144 -DAsyncLoggerConfig.RingBufferSize=262144 -Dlog4j2.AsyncQueueFullPolicy=Discard -DAppPID -jar hsrcm-wfl-server-202401.09.03-small.jar jfile=config/application.properties
root 481753 0.2 1.2 10371968 851904 ? Sl 2024 644:22 java -jar -Dloader.path=./lib -Xms2048m -Xmx2048m -Duname=aarch64 -DAppPID -DDUBBO_IP_TO_REGISTRY=10.6.7.162 permitcenter.jar
root 672640 0.4 0.8 10950080 564800 ? Sl 2024 491:35 java -Dunisql.dll.dir= -Dloader.path=./lib -Xmx512m -Xms512m -XX:+HeapDumpOnOutOfMemoryError -DAsyncLogger.RingBufferSize=262144 -DAsyncLoggerConfig.RingBufferSize=262144 -Dlog4j2.AsyncQueueFullPolicy=Discard -jar hsrcm-messager-server-202401.09.00-small.jar jfile=config/application.properties
root 684 0.0 0.4 333248 264256 ? Ss 2024 44:36 /usr/lib/systemd/systemd-journald
root 4146954 0.5 0.2 6150912 174016 ? Sl Feb03 12:56 /opt/qaxsafe/qaxsafed
root 331199 0.0 0.1 874624 88320 ? Sl 09:41 0:01 nginx: manager process
root 2141590 0.0 0.1 1418944 75136 ? Sl 2024 52:30 ./bin/easy_process_sampler
root 1496 0.0 0.1 680448 74752 ? Ssl 2024 46:38 /usr/sbin/rsyslogd -n -iNONE
According to online resources, this <span>auditd</span> is an auditing process of Kylin, which has a memory leak issue.
Audit Memory Leak Analysis Report · Kylin Server Operating System · See Cloud
Upgrade and Fix
Official Reference: Kylin Advanced Server Operating System V10 SP1-mate-indicators Memory Usage Issue Handling.pdf
Kylin Memory Leak Upgrade Plan.docx
Relevant upgrade packages have been uploaded to Alibaba Cloud Drive.
Before the upgrade: memory usage was 54G, of which the <span>auditd</span> process accounted for <span>43%</span>.
[root@kylinV10arm161 rpm]# free -h
total used free shared buff/cache available
Mem: 62Gi 54Gi 4.8Gi 2.3Gi 3.7Gi 550Mi
Swap: 15Gi 1.1Gi 14Gi
Upgrade Process
[root@kylinV10arm161 rpm]# rpm -qa | grep mate-indicators
mate-indicators-20150918kord0ukui58-10.p05.ky10.aarch64
[root@kylinV10arm161 rpm]# rpm -qa | grep audit
audit-3.0-5.se.06.ky10.aarch64
audit-libs-3.0-5.se.06.ky10.aarch64
python3-audit-3.0-5.se.06.ky10.aarch64
[root@kylinV10arm161 rpm]# rpm -Uvh *.rpm
warning: audit-3.0-5.se.12.ky10.aarch64.rpm: Header V4 RSA/SHA1 Signature, key ID 7a486d9f: NOKEY
Verifying... ################################# [100%]
Preparing... ################################# [100%]
Updating / installing...
1:audit-libs-3.0-5.se.12.ky10 ################################# [ 9%]
2:audit-3.0-5.se.12.ky10 ################################# [ 18%]
/sbin/ldconfig: /usr/lib64/libLLVM-7.so is not a symbolic link
3:python3-audit-3.0-5.se.12.ky10 ################################# [ 27%]
4:rasdaemon-0.6.6-1.p03.ky10 ################################# [ 36%]
5:mate-indicators-20150918kord0ukui################################# [ 45%]
Warning: Schema “org.gnome.crypto.cache” has path “/desktop/gnome/crypto/cache/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.crypto.pgp” has path “/desktop/gnome/crypto/pgp/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.locale” has path “/system/locale/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy” has path “/system/proxy/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.http” has path “/system/proxy/http/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.https” has path “/system/proxy/https/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.ftp” has path “/system/proxy/ftp/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.socks” has path “/system/proxy/socks/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
6:audit-help-3.0-5.se.12.ky10 ################################# [ 55%]
Cleaning up / removing...
7:python3-audit-3.0-5.se.06.ky10 ################################# [ 64%]
8:audit-3.0-5.se.06.ky10 ################################# [ 73%]
/sbin/ldconfig: /usr/lib64/libLLVM-7.so is not a symbolic link
9:audit-libs-3.0-5.se.06.ky10 ################################# [ 82%]
10:rasdaemon-0.6.6-1.ky10 ################################# [ 91%]
11:mate-indicators-20150918kord0ukui################################# [100%]
/sbin/ldconfig: /usr/lib64/libLLVM-7.so is not a symbolic link
/sbin/ldconfig: /usr/lib64/libLLVM-7.so is not a symbolic link
[root@kylinV10arm161 rpm]# rpm -qa | grep mate-indicators
mate-indicators-20150918kord0ukui58-10.p07.ky10.aarch64
[root@kylinV10arm161 rpm]# rpm -qa | grep audit
audit-libs-3.0-5.se.12.ky10.aarch64
audit-3.0-5.se.12.ky10.aarch64
audit-help-3.0-5.se.12.ky10.noarch
python3-audit-3.0-5.se.12.ky10.aarch64
After the upgrade: memory usage decreased by half, but the <span>mate-indicators</span> process reappeared.
[root@kylinV10arm161 rpm]# systemctl daemon-reload
[root@kylinV10arm161 rpm]# ps aux --sort=-%mem |head -20
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2884 0.1 28.7 20328640 18888832 ? Sl 2024 558:25 /usr/local/lib64/mate-indicators/libexec/mate-indicators
# Application's Java program memory omitted...
[root@kylinV10arm161 rpm]# free -h
total used free shared buff/cache available
Mem: 62Gi 26Gi 32Gi 2.3Gi 3.8Gi 28Gi
Swap: 15Gi 1.1Gi 14Gi
Mitigation and Solutions for mate-indicators Component Memory Usage
Looking at the memory usage rate again
[root@kylinV10arm161 rpm]# ps aux --sort=-%mem |head
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 2884 0.1 28.7 20329600 18889920 ? Sl 2024 558:26 /usr/local/lib64/mate-indicators/libexec/mate-indicators<span>mate-indicators</span> should also disable graphical display.
[root@kylinV10arm161 rpm]# systemctl set-default multi-user.target
Removed /etc/systemd/system/default.target.
Created symlink /etc/systemd/system/default.target → /usr/lib/systemd/system/multi-user.target.
[root@kylinV10arm161 rpm]# systemctl mask lightdm.service
Created symlink /etc/systemd/system/lightdm.service → /dev/null.10.xx.xx.xxx1 upgrade + disable, completed ✔
10.xx.xx.xxx2 upgrade + disable, completed ✔
Reloading Services
Reload the service <span>systemctl daemon-reload</span>
It is also best to restart the machine by executing the command:<span>init 6</span>
Conclusion
Kylin V10 SP1/SP2 needs to be upgraded; SP3 has no issues.
The content of this article is a record of my learning, and my knowledge level is limited; it is only for documentation purposes~
