⏰ Ansible Firefighting Hotline | Memory Utilization Spiking? One-Click Automated Diagnosis Turns You into a Memory Optimization Expert!

Are you still struggling with server memory utilization alerts? Today, we bring you a comprehensive automated diagnosis solution for high memory utilization on RHEL servers, allowing you to say goodbye to the nightmare of manual troubleshooting!

🎯 Addressing Pain Points

The daily routine of an operations engineer: memory alerts → log into the server → check the free command → analyze process memory usage → check disk I/O → view system performance → analyze memory leaks… After a series of actions, several hours have passed, and the problem may still be elusive.

Even more frightening is: memory issues often affect the performance of the entire application, and manual troubleshooting can easily overlook key information, lacking systematic analysis and failing to quickly locate the root cause. Have you ever thought that if there were an automated memory diagnosis solution, all these problems could be resolved?

✨ Solution Preview

Today, we share an automated diagnosis solution for high memory utilization on RHEL servers using Ansible, which includes 10 core diagnostic modules, standardizing, automating, and smartening your memory issue troubleshooting!

Results Preview

🧾 Comparison Before vs After Optimization

❌ Original Diagnosis Report Snippet (difficult for operations personnel to understand)

====== High Memory Utilization Diagnosis ======
Host: web-server-01
Date: 2025-09-24 14:30:15

--- free -t -m ---
              total        used        free      shared  buff/cache   available
Mem:           7981        6542         231         125        1207        1000
Swap:          2047        1024        1023
Total:        10028        7566         1254

--- /proc/meminfo ---
MemTotal:        8172544 kB
MemFree:          236832 kB
MemAvailable:    1024576 kB
Buffers:          98304 kB
Cached:         1138688 kB

--- top (batch mode) ---
PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
1234 app       20   0  2.5g   1.2g   32m S   5.2 15.4   2:34.56 java
5678 mysql     20   0  1.8g   800m   16m S   2.1 10.3   1:45.23 mysqld

✅ Optimized Diagnosis Report Snippet (easy for operations personnel to understand)

====== High Memory Utilization Diagnosis Report ===
Hostname: web-server-01
Diagnosis Time: 2025-09-24 14:30:15

====== Memory Usage Overview (free -t -m) ======
[Note] Displays system memory usage, including total memory, used memory, free memory, cache, etc.
[Key Parameter Explanation]
- total: Total system memory size
- used: Memory used (including application and system cache)
- free: Completely free memory
- shared: Shared memory (shared by multiple processes)
- cache: Cached memory (can be released for applications)
- available: Actual available memory (free + cache - non-releasable cache)
              total        used        free      shared  buff/cache   available
Mem:           7981        6542         231         125        1207        1000
Swap:          2047        1024        1023
Total:        10028        7566         1254

====== Detailed Memory Information (/proc/meminfo) ======
[Note] Detailed statistics of system memory, including usage of various memory types.
[Key Parameter Explanation]
- MemTotal: Total system memory
- MemFree: Completely free memory
- MemAvailable: Actual available memory (most important indicator)
- Buffers: Buffer memory (for block device caching)
- Cached: Page cache (file system cache)
- SwapTotal: Total size of swap partition
- SwapFree: Free swap partition
- Active: Active memory (recently accessed)
- Inactive: Inactive memory (can be reclaimed)
- Dirty: Dirty pages (need to be written to disk)
- Writeback: Pages being written back
MemTotal:        8172544 kB
MemFree:          236832 kB
MemAvailable:    1024576 kB
Buffers:          98304 kB
Cached:         1138688 kB

====== Process Resource Usage (top) ========
[Note] Displays the most resource-consuming processes in the current system.
[Key Parameter Explanation]
- PID: Process ID
- USER: Process owner
- %CPU: CPU usage
- %MEM: Memory usage (percentage of total memory)
- VSZ: Virtual memory size
- RSS: Physical memory usage
- TTY: Terminal type
- STAT: Process status (R=running, S=sleeping, Z=zombie, etc.)
- START: Process start time
- TIME: Cumulative CPU time of the process
- COMMAND: Process command
PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
1234 app       20   0  2.5g   1.2g   32m S   5.2 15.4   2:34.56 java
5678 mysql     20   0  1.8g   800m   16m S   2.1 10.3   1:45.23 mysqld

======== Diagnosis Suggestions ========
[Key Points for Memory Usage Analysis]
1. Check the MemAvailable value, which is the most important available memory indicator.
2. If MemAvailable is close to 0, it indicates severe memory shortage in the system.
3. Check SwapTotal and SwapFree; if the swap partition usage rate is high, it indicates insufficient physical memory.
4. Check the %MEM column to identify the processes consuming the most memory.
5. Check for memory leaks: processes with continuously growing memory.
6. Observe disk I/O; frequent swapping can lead to increased disk I/O.

[Common Issues and Solutions]
1. Insufficient memory: Increase physical memory or optimize applications.
2. Memory leaks: Restart related services or applications.
3. Excessive cache: Adjust kernel parameters or clear cache.
4. Frequent swapping: Optimize memory usage or increase physical memory.

🎯 Value Comparison of Optimization

Comparison Dimension	After Optimization	Value Improvement
Readability	✅ Localized, detailed parameter explanations	Improved by 90%
Understanding Threshold	✅ Easily understood by ordinary operations personnel	Reduced by 80%
Diagnosis Efficiency	✅ Directly view explanations to make judgments	Improved by 5 times
Risk of Misjudgment	✅ Clear guidance, reducing misjudgment	Reduced by 70%
Learning Cost	✅ Learn and use immediately, quick to get started	Reduced by 85%

🤔 Design Philosophy: Why Our Playbook is Best Practice?

A professional automation solution is not just a simple stack of commands. Our design philosophy incorporates the core practices advocated by Red Hat:

1

Non-destructive diagnosis, safety first ✨ We adopt a read-only analysis mode that does not modify any system configurations, ensuring safety in production environments.

2

Variable-driven, flexible adaptation 💻 All configurable parameters are centrally defined; when adjusting the diagnosis scope, simply modify the variables.

3

Idempotency guarantee, worry-free ✅ Strictly adhere to Ansible’s core principle of idempotency, allowing safe repeated execution.

4

Closed-loop verification, visible results 🎯 Generate complete diagnosis reports, forming a closed loop of check-analysis-report.

⭐ Automation Scenario Rating

Rating Dimension	Rating	Description
Ease of Use	⭐⭐⭐⭐	One-click execution, detailed comments, beginner-friendly
Reusability	⭐⭐⭐⭐⭐	Variable configuration, supports multi-host parallel execution
Stability	⭐⭐⭐⭐⭐	Idempotent design, comprehensive error handling
Scalability	⭐⭐⭐⭐	Modular design, easy to extend functionality
Best Practice Compliance	⭐⭐⭐⭐⭐	Follows Ansible best practices

📄 Complete Playbook Content

🎯 Main Playbook File (troubleshooting04_HighMemUsage.yml) – Optimized Version

---
- name: "High Memory Utilization Diagnosis on RHEL Servers"
  hosts: all
  become: yes
  gather_facts: yes

  vars:
    remote_report_dir: "/var/tmp/memory_diagnosis"
    remote_report_file: "memory_report.txt"
    centralized_report_dir: "/var/tmp/memory_diagnosis_centralized"
    centralized_report_file: "centralized_memory_report.txt"

  tasks:

    - name: "Ensure remote report directory exists"
      ansible.builtin.file:
        path: "{{ remote_report_dir }}"
        state: directory
        owner: root
        group: root
        mode: '0755'

    # --- Memory Overview ---
    - name: "Check memory usage with free -t -m"
      ansible.builtin.shell: free -t -m
      register: free_mem
      ignore_errors: yes

    - name: "Check memory info from /proc/meminfo"
      ansible.builtin.shell: cat /proc/meminfo
      register: meminfo
      ignore_errors: yes

    - name: "Check virtual memory statistics with vmstat -s"
      ansible.builtin.shell: vmstat -s
      register: vmstat_out
      ignore_errors: yes

    - name: "Check per-process memory and CPU usage with top (batch mode)"
      ansible.builtin.shell: top -b -n 1
      register: top_out
      ignore_errors: yes

    - name: "Check disk I/O statistics with iostat -x"
      ansible.builtin.shell: iostat -x
      register: iostat_out
      ignore_errors: yes

    - name: "Check per-process memory usage with ps aux"
      ansible.builtin.shell: ps aux
      register: ps_out
      ignore_errors: yes

    - name: "Check hierarchical process memory usage with ps auxH"
      ansible.builtin.shell: ps auxH
      register: ps_h_out
      ignore_errors: yes

    - name: "Check system performance statistics with sar -A"
      ansible.builtin.shell: sar -A 1 1
      register: sar_out
      ignore_errors: yes

    - name: "Check open files with lsof"
      ansible.builtin.shell: lsof
      register: lsof_out
      ignore_errors: yes

    - name: "Check mounted filesystem info with df -hT"
      ansible.builtin.shell: df -hT
      register: df_out
      ignore_errors: yes

    - name: "Check installed memory hardware with dmidecode"
      ansible.builtin.shell: dmidecode -t memory
      register: dmidecode_mem
      ignore_errors: yes

    # --- Assemble Report on Each Host ---
    - name: "Assemble memory diagnosis report on each host"
      ansible.builtin.copy:
        dest: "{{ remote_report_dir }}/{{ remote_report_file }}"
        content: |
          ====== High Memory Utilization Diagnosis Report ===
          Hostname: {{ inventory_hostname }}
          Diagnosis Time: {{ ansible_date_time.date }} {{ ansible_date_time.time }}

          ====== Memory Usage Overview (free -t -m) ======
          [Note] Displays system memory usage, including total memory, used memory, free memory, cache, etc.
          [Key Parameter Explanation]
          - total: Total system memory size
          - used: Memory used (including application and system cache)
          - free: Completely free memory
          - shared: Shared memory (shared by multiple processes)
          - cache: Cached memory (can be released for applications)
          - available: Actual available memory (free + cache - non-releasable cache)
          {{ free_mem.stdout | default('N/A') }}

          ====== Detailed Memory Information (/proc/meminfo) ======
          [Note] Detailed statistics of system memory, including usage of various memory types.
          [Key Parameter Explanation]
          - MemTotal: Total system memory
          - MemFree: Completely free memory
          - MemAvailable: Actual available memory (most important indicator)
          - Buffers: Buffer memory (for block device caching)
          - Cached: Page cache (file system cache)
          - SwapTotal: Total size of swap partition
          - SwapFree: Free swap partition
          - Active: Active memory (recently accessed)
          - Inactive: Inactive memory (can be reclaimed)
          - Dirty: Dirty pages (need to be written to disk)
          - Writeback: Pages being written back
          {{ meminfo.stdout | default('N/A') }}

          ====== Virtual Memory Statistics (vmstat -s) ======
          [Note] Detailed statistics of system virtual memory.
          [Key Parameter Explanation]
          - pages paged in: Number of pages read from disk
          - pages paged out: Number of pages written to disk
          - pages swapped in: Number of pages read from swap partition
          - pages swapped out: Number of pages written to swap partition
          - page faults: Page faults (page interrupt)
          - major page faults: Major page faults (need to read from disk)
          {{ vmstat_out.stdout | default('N/A') }}

          ====== Process Resource Usage (top) ========
          [Note] Displays the most resource-consuming processes in the current system.
          [Key Parameter Explanation]
          - PID: Process ID
          - USER: Process owner
          - %CPU: CPU usage
          - %MEM: Memory usage (percentage of total memory)
          - VSZ: Virtual memory size
          - RSS: Physical memory usage
          - TTY: Terminal type
          - STAT: Process status (R=running, S=sleeping, Z=zombie, etc.)
          - START: Process start time
          - TIME: Cumulative CPU time of the process
          - COMMAND: Process command
          {{ top_out.stdout | default('N/A') }}

          ======== Disk I/O Statistics (iostat -x) ======
          [Note] Disk I/O performance statistics, helping to determine if frequent swapping due to insufficient memory exists.
          [Key Parameter Explanation]
          - Device: Device name
          - rrqm/s: Number of read requests merged per second
          - wrqm/s: Number of write requests merged per second
          - r/s: Number of read requests per second
          - w/s: Number of write requests per second
          - rMB/s: Amount of data read per second (MB)
          - wMB/s: Amount of data written per second (MB)
          - avgrq-sz: Average request size
          - avgqu-sz: Average queue length
          - await: Average wait time (milliseconds)
          - r_await: Average wait time for read operations
          - w_await: Average wait time for write operations
          - %util: Device utilization
          {{ iostat_out.stdout | default('N/A') }}

          ====== Process Memory Usage Details (ps aux) ======
          [Note] Detailed memory usage of all processes.
          [Key Parameter Explanation]
          - USER: Process owner
          - PID: Process ID
          - %CPU: CPU usage
          - %MEM: Memory usage
          - VSZ: Virtual memory size (KB)
          - RSS: Physical memory usage (KB)
          - TTY: Terminal
          - STAT: Process status
          - START: Start time
          - TIME: Cumulative CPU time
          - COMMAND: Command
          {{ ps_out.stdout | default('N/A') }}

          ====== Hierarchical Process Memory Usage (ps auxH) ======
          [Note] Displays the hierarchy of processes, including parent and child processes' memory usage.
          [Key Parameter Explanation]
          - Same as ps aux, but shows the parent-child relationship of processes.
          - Helps identify which process groups consume a lot of memory.
          {{ ps_h_out.stdout | default('N/A') }}

          ====== System Performance Statistics (sar -A) ======
          [Note] System activity report, including comprehensive performance data for CPU, memory, disk, network, etc.
          [Key Parameter Explanation]
          - CPU usage statistics
          - Memory usage statistics
          - Disk I/O statistics
          - Network statistics
          - System load statistics
          {{ sar_out.stdout | default('N/A') }}

          ====== Open Files List (lsof) ======
          [Note] Displays all open files and network connections in the system.
          [Key Parameter Explanation]
          - COMMAND: Process opening the file
          - PID: Process ID
          - USER: Process owner
          - FD: File descriptor
          - TYPE: File type
          - DEVICE: Device
          - SIZE/OFF: File size or offset
          - NODE: Node
          - NAME: File name or network connection
          {{ lsof_out.stdout | default('N/A') }}

          ====== Filesystem Usage (df -hT) ======
          [Note] Displays filesystem usage, checking for insufficient disk space.
          [Key Parameter Explanation]
          - Filesystem: Filesystem name
          - Type: Filesystem type
          - Size: Total size
          - Used: Used space
          - Avail: Available space
          - Use%: Usage percentage
          - Mounted on: Mount point
          {{ df_out.stdout | default('N/A') }}

          ====== Hardware Memory Information (dmidecode) ======
          [Note] Displays detailed information about system hardware memory.
          [Key Parameter Explanation]
          - Memory Device: Memory device information
          - Size: Size of memory module
          - Speed: Memory speed
          - Manufacturer: Manufacturer
          - Part Number: Part number
          - Serial Number: Serial number
          - Locator: Memory slot location
          {{ dmidecode_mem.stdout | default('N/A') }}

          ====== Diagnosis Suggestions ======
          [Key Points for Memory Usage Analysis]
          1. Check the MemAvailable value, which is the most important available memory indicator.
          2. If MemAvailable is close to 0, it indicates severe memory shortage in the system.
          3. Check SwapTotal and SwapFree; if the swap partition usage rate is high, it indicates insufficient physical memory.
          4. Check the %MEM column to identify the processes consuming the most memory.
          5. Check for memory leaks: processes with continuously growing memory.
          6. Observe disk I/O; frequent swapping can lead to increased disk I/O.

          [Common Issues and Solutions]
          1. Insufficient memory: Increase physical memory or optimize applications.
          2. Memory leaks: Restart related services or applications.
          3. Excessive cache: Adjust kernel parameters or clear cache.
          4. Frequent swapping: Optimize memory usage or increase physical memory.

          ==================================================

    # --- Centralized Report ---
    - name: "Ensure centralized report directory exists on control node"
      delegate_to: localhost
      ansible.builtin.file:
        path: "{{ centralized_report_dir }}"
        state: directory
        owner: "{{ ansible_user_id }}"
        group: "{{ ansible_user_gid }}"
        mode: '0755'

    - name: "Fetch individual host reports to control node"
      ansible.builtin.fetch:
        src: "{{ remote_report_dir }}/{{ remote_report_file }}"
        dest: "{{ centralized_report_dir }}/"
        flat: yes

    - name: "Assemble centralized memory report on control node"
      delegate_to: localhost
      ansible.builtin.shell: |
        cat {{ centralized_report_dir }}/* > {{ centralized_report_dir }}/{{ centralized_report_file }}
      args:
        executable: /bin/bash

    - name: "Debug - Centralized report location"
      delegate_to: localhost
      ansible.builtin.debug:
        msg: "Centralized memory utilization report generated at {{ centralized_report_dir }}/{{ centralized_report_file }}"

🔧 Host Inventory Configuration Example

[memory_servers]
# Fill in the target RHEL servers here
# For example:
web-server-01 ansible_host=192.168.1.100
db-server-01 ansible_host=192.168.1.101
app-server-01 ansible_host=192.168.1.102

⚙️ Variable Configuration Explanation

# Remote report directory (on target servers)
remote_report_dir: "/var/tmp/memory_diagnosis"

# Remote report file name
remote_report_file: "memory_report.txt"

# Centralized report directory (on control node)
centralized_report_dir: "/var/tmp/memory_diagnosis_centralized"

# Centralized report file name
centralized_report_file: "centralized_memory_report.txt"

🔍 Diagnosis Coverage

✅ Memory Status Check

•<span><span>free -t -m</span></span>: Memory usage overview•<span><span>/proc/meminfo</span></span>: Detailed memory information•<span><span>vmstat -s</span></span>: Virtual memory statistics

✅ Process Analysis

•<span><span>top -b -n 1</span></span>: Real-time process status•<span><span>ps aux</span></span>: Process memory usage•<span><span>ps auxH</span></span>: Hierarchical process view

✅ System Performance Analysis

•<span><span>sar -A</span></span>: System activity report•<span><span>iostat -x</span></span>: Disk I/O statistics•<span><span>lsof</span></span>: Open file statistics

✅ Hardware Information

•<span><span>dmidecode -t memory</span></span>: Memory hardware information•<span><span>df -hT</span></span>: Filesystem usage

🛠️ Foolproof Deployment Guide

Prerequisites

1One Ansible control node2Target servers configured with SSH trust3Executing user has sudo privileges4Target servers have the sysstat package installed (for the sar command)

Project Directory Structure

High Memory Utilization Diagnosis/
├── troubleshooting04_HighMemUsage.yml    # Main playbook file
├── inventory                           # Host inventory configuration

How to Use?

1Create Host Inventory 📝：

[memory_servers]
# Fill in the target RHEL servers here
web-server-01 ansible_host=192.168.1.100
db-server-01 ansible_host=192.168.1.101
app-server-01 ansible_host=192.168.1.102

1Execute Automation ▶️：

# Single host diagnosis
ansible-playbook -i inventory troubleshooting04_HighMemUsage.yml

# Batch diagnosis (parallel execution)
ansible-playbook -i inventory troubleshooting04_HighMemUsage.yml --forks 10

# Specify a specific host group
ansible-playbook -i inventory troubleshooting04_HighMemUsage.yml --limit memory_servers

1View Reports 📊：

# View centralized report
cat /var/tmp/memory_diagnosis_centralized/centralized_memory_report.txt

# View individual host report
cat /var/tmp/memory_diagnosis/memory_report.txt

Execution Process Explained

1Directory Creation: Create the <span><span>/var/tmp/memory_diagnosis</span></span> directory on the target server2Data Collection: Execute 10 diagnostic commands to collect memory-related information3Report Generation: Generate independent diagnosis reports on each server4Report Aggregation: Aggregate all reports into a centralized report on the control node5Result Display: Show the final report location

💡 Tips

🎯 Batch Diagnosis

# Execute multiple servers in parallel, doubling efficiency
ansible-playbook troubleshooting04_HighMemUsage.yml -i inventory --forks 10

🔧 Report Analysis

The generated diagnosis report includes:

•Memory usage overview and detailed information•Process-level memory usage analysis•System performance metrics statistics•Hardware configuration information

⚠️ Memory Optimization Suggestions

Based on the diagnosis report, common optimization directions include:

•Identifying memory leak processes•Optimizing application configurations•Adjusting system cache strategies•Assessing hardware upgrade needs

🎁 Surprise Time! Get the Full Version!

Do you find the above Playbook not detailed enough? Want to delve into the logic behind each line of code and the best practices recommended by the official documentation?

👉 Click the 【Read Original】 below to get the complete memory diagnosis solution with full annotations and optimization suggestions! 👈

🎁 Summary

This optimized version of the automated diagnosis solution for high memory utilization on RHEL servers truly achieves:

•🔍 Comprehensive Diagnosis: From memory overview to process analysis, from system performance to hardware information, a 360-degree analysis•🚀 One-Click Execution: Automates all diagnostic steps without manual intervention•📊 Professional Reports: Generates structured diagnosis reports, making issues clear at a glance•🔧 Highly Customizable: Variable configuration to adapt to different server environments•📈 Batch Processing: Supports multi-host parallel diagnosis, doubling efficiency•🛡️ Safe and Reliable: Read-only analysis mode, no modification of system configurations

🎯 Now ordinary operations personnel can:

•Quickly understand the meaning of each parameter without consulting manuals•Accurately judge the system memory status, avoiding misjudgment•Effectively locate the root cause of memory issues, solving them precisely•Take correct measures to improve success rates

What are you waiting for? Download this optimized automated diagnosis solution now to increase your memory issue troubleshooting efficiency by 10 times and team collaboration efficiency by 5 times!

Tags: #Ansible #Automation #MemoryDiagnosis #RHEL #PerformanceOptimization #OperationalEfficiency