🎯 Classic Practical Cases | Analysis of 6 Major Boot Failure Scenarios, from Kernel Crashes to File System Corruption!

📖 Introduction: Do You Feel This Way Too?

First Paragraph: Pain Point Scenario

Have you ever encountered such a desperate situation? After a system update, it fails to boot, displaying “Kernel panic – not syncing: VFS: Unable to mount root fs,” and you have no idea what to do. Worse, you try to boot from an old kernel, but the system keeps rebooting, and you don’t even know where the problem lies!

You might encounter:

•New kernel crashes after system update, but you don’t know how to boot from the old kernel•System shows “Out of memory” error during boot, unable to complete startup•After system update, it shows “Unable to mount root fs on unknown-block(0,0)”•System enters maintenance mode after boot, asking for root password•System displays file system corruption traceback information during boot•System keeps rebooting during the boot process, stuck in a reboot loop

You try various methods but cannot find the root cause of the problem. Your boss asks you, “Why can’t the system boot?” You can only helplessly reply, “There was a problem after the update, but I don’t know the specific reason…” This vicious cycle of “boot failure after update, with no way to diagnose” is driving you crazy, isn’t it?

If you have also experienced this nightmare of “boot failure after system update with no diagnosis,” then today FYC has good news for you:Red Hat has officially provided 6 classic practical cases! These cases cover all common scenarios from kernel crashes to file system corruption. Just follow the case operations, and you can quickly solve the problems!

Second Paragraph: Overview of Solutions

Today we will discussRed Hat’s official classic practical cases, which summarize the essence of diagnosing boot failures in RHEL systems. We will start with 6 practical scenarios, guiding you step by step on how to handle common boot issues such as kernel crashes, memory shortages, file system corruption, and maintenance mode.

This article will bring you:

•🎯 Case 1: Booting from Backup Kernel: What to do when the new kernel crashes? How to boot from the old kernel?•🔧 Case 2: Huge Page Configuration Error: What to do when OOM prevents startup? How to fix it without using ISO?•✅ Case 3: Missing initramfs: What to do when unable to mount the root file system after an update?•⚠️ Case 4: Maintenance Mode: What to do when the system enters maintenance mode after boot?•💡 Case 5: File System Repair: What to do when the file system is corrupted? How to repair it in rescue mode?•🚨 Case 6: Insufficient Audit Log Space: What to do when the system shuts down during the boot process?

Follow FYC, and say goodbye to the era of “boot failure after system update with no diagnosis!”

Third Paragraph: 5-Dimension Scoring Table

Dimension	Score	Description
Difficulty Level	⭐⭐⭐⭐	Requires in-depth understanding of the boot process and troubleshooting methods
Practical Value	⭐⭐⭐⭐⭐	Solves key issues of boot failure after system updates
Technical Depth	⭐⭐⭐⭐	Involves kernel, memory management, file systems, LVM, etc.
Operability	⭐⭐⭐⭐⭐	All commands and diagnostic steps can be used directly
Urgency	⭐⭐⭐⭐⭐	System boot failure is usually the highest priority issue, affecting a wide range

📚 Main Content: Packed with Useful Information, But Needs to Be “Fed to You”!

⚡ 1. Case 1: What to Do When the New Kernel Crashes? A Complete Guide to Booting from Backup Kernel

📋 Problem Scenario

After a system update, a new kernel is installed, but it causes the system to crash and fail to boot. You need to boot the system from the old kernel.

🎯 Solution: Temporarily Boot from Backup Kernel

Step 1: Enter GRUB Menu

# 1. When the system starts, a message similar to the following will be displayed: 
# "Press any key to enter the menu
#  Booting Red Hat Enterprise Linux (3.10.0-1062.4.1.el7.x86_64) in 3 seconds..."

# 2. During the countdown, press any key to enter the GRUB menu

Step 2: Select Backup Kernel

# 1. In the GRUB menu, use the arrow keys to select the previous kernel version
# 2. After highlighting the old kernel, press Enter to boot
# 3. The system will boot using the old kernel

Example of GRUB Menu:

Red Hat Enterprise Linux Server (3.10.0-1062.4.1.el7.x86_64)  ← New Kernel (Problematic)
Red Hat Enterprise Linux Server (3.10.0-1062.1.1.el7.x86_64)  ← Old Kernel (Select this)
Red Hat Enterprise Linux Server (3.10.0-957.el7.x86_64)        ← Older Kernel

🔧 Solution: Permanently Set Default Kernel

If the new kernel continues to have issues, you can permanently set the default kernel:

Reference Document:

•How do I change the default kernel in GRUB that is loaded at startup?

📊 Diagnostic Steps

# 1. Check the current kernel version
uname -r

# 2. Check yum logs for recent kernel updates
grep kernel /var/log/yum.log | tail -20

# 3. Check the default kernel in GRUB configuration
# RHEL 6 and below:
cat /boot/grub/grub.conf | grep default

# RHEL 7+:
cat /boot/grub2/grub.cfg | grep "menuentry"

💡 Root Cause

•Running <span><span>yum update</span></span>may install a new kernel, which becomes the default kernel•The new kernel may be incompatible with hardware or drivers, causing the system to crash•The kernel provided by Red Hat will automatically be added to the boot loader, but you can still boot the previously installed kernels

⚠️ Preventive Measures

If you do not want <span><span>yum update</span></span>to update or install a new kernel, you can refer to:

•How do I exclude updating the kernel or other packages in RHEL 5/6 while updating system via yum?

🔧 2. Case 2: Huge Page Configuration Error Causes OOM Preventing Startup (Without Using ISO Fix)

📋 Problem Scenario

During system startup, an “Out of memory” error is displayed, preventing completion of the startup. This is usually because the huge page (hugepages) configuration value exceeds the total system memory. You need to fix the issue without using ISO.

🎯 Scenario 1: Huge Pages Set in Kernel Command Line

Problem Diagnosis:

# 1. In the GRUB menu, press 'e' to edit the boot entry
# 2. Check the kernel command line parameters and find:
# hugepagesz=2M hugepages=6000
# Total system memory: 10GiB
# Calculation: 2M * 6000 = 12GiB > 10GiB (exceeds total memory!)

Solution:

# 1. Boot the system, stop at the GRUB menu
# 2. Press 'e' to edit the boot entry
# 3. Find the line containing hugepages and delete the hugepages related parameters
#    Delete: hugepagesz=2M hugepages=6000
# 4. Press Ctrl+X to continue booting
# 5. After the system boots, calculate the correct number of huge pages and modify the GRUB configuration

# Calculation Example: If the system has 10GiB RAM, and huge page size is 2MiB
# The number of huge pages should not exceed 4500 (approximately 8.78GiB)
echo "2*4500/2^10" | bc -l
# Output: 8.78906250000000000000

# Modify the GRUB configuration to use the correct number of huge pages
grub2-mkconfig -o /etc/grub2.cfg

🎯 Scenario 2: Huge Pages Set in Configuration File (RHEL 7)

Solution:

# ======== Step 1: Bypass sysctl service ========
# 1. Restart the system, stop at the GRUB menu
# 2. Press 'e' to edit the boot entry
# 3. Find the line starting with linux16 or linuxefi
# 4. Add the following parameters:
systemd.mask=systemd-sysctl.service rd.systemd.unit=emergency.target systemd.mask=tuned.service

# 5. Press Ctrl+X to continue booting
# 6. The system will enter emergency mode (dracut shell)

# ======== Step 2: Fix Configuration in Emergency Mode ======
# The system will display:
# [    2.906661] systemd[1]: Started Emergency Shell.
# [  OK  ] Started Emergency Shell.
# [    2.909506] systemd[1]: Reached target Emergency Mode.
# [  OK  ] Reached target Emergency Mode.
# Generating "/run/initramfs/rdsosreport.txt"
# Entering emergency mode. Exit the shell to continue.
# Type "journalctl" to view system logs.
# :/# 

# 7. Find files containing nr_hugepages
grep -r nr_hugepages /etc/* -l

# 8. Edit the file containing huge page configuration (using vi):
# /etc/sysctl.conf
# /etc/sysctl.d/*.conf
# /usr/lib/sysctl.d/*.conf
# /lib/sysctl.d/*.conf
# /usr/local/lib/sysctl.d/*.conf

# 9. Delete or comment out the huge page configuration line
# 10. Exit dracut shell: exit

# ====== Step 3: Complete the Fix After System Boots ========
# 11. After the system boots, execute the following steps:
# Set huge pages to 0
echo 0 > /proc/sys/vm/nr_hugepages

# Modify the configuration file, comment out or delete the erroneous huge page settings
# Rebuild initramfs
dracut -f

# 12. Restart the system to verify
reboot

🔧 Root Cause

•The huge page configuration value exceeds the total system memory, causing the system to run out of memory (OOM) during startup•Huge pages are applied during the initramfs phase or systemd startup, preventing the system from completing the boot

✅ 3. Case 3: Unable to Mount Root File System After Update (Missing initramfs)

📋 Problem Scenario

After a system update, it fails to boot, displaying the following error:

VFS: Cannot open root device XXX or unknown-block(0,0)
Please append a correct "root=" boot option; here are the available partitions:
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

🎯 Solution: 7-Step Complete Repair Process

Step 1: Boot from Old Kernel

# 1. Wait for the "Booting Red Hat Enterprise Linux Server in X seconds" countdown
# 2. Press any key to enter the GRUB menu
# 3. Use the arrow keys to select the old kernel and press Enter to boot

# Note: If no old kernel is available, enter rescue mode, then chroot to the root file system and continue to Step 2

Step 2: Check /boot and /tmp Space

# Check the space of /boot and /tmp partitions (RHEL 7/8 check /boot and /var/tmp)
df -h /boot /tmp

# If space is nearly full, check the installed kernel versions
# RHEL 6/7:
rpm --last -q kernel

# RHEL 8/9:
rpm --last -q kernel-core kernel

# Remove the oldest kernel to free up space
yum remove kernel-<oldest-version>

Step 3: Remove and Reinstall Latest Kernel

# RHEL 6 & 7:
yum remove kernel-<newversion>-<release>.<arch>
yum install kernel-<newversion>-<release>.<arch>

# RHEL 8 & 9:
yum remove kernel-core-<newversion>-<release>.<arch> kernel-<newversion>-<release>.<arch>
yum install kernel-core-<newversion>-<release>.<arch> kernel-<newversion>-<release>.<arch>

# Warning: When removing and reinstalling kernels, it is recommended to keep at least one known good kernel as a backup

Step 4: Verify initramfs File Has Been Created

# Check the /boot directory
ls -l /boot

# You should see the corresponding initramfs file for the kernel:
# initramfs-<kernel-version>.img

Step 5: Check initramfs Declaration in GRUB Configuration

# RHEL 6 and below:
cat /boot/grub/grub.conf | grep -A 5 "<kernel-version>"

# RHEL 7 (BIOS):
cat /boot/grub2/grub.cfg | grep -A 5 "<kernel-version>"

# RHEL 7 (UEFI):
cat /boot/efi/EFI/redhat/grub.cfg | grep -A 5 "<kernel-version>"

# RHEL 8/9:
cat /boot/loader/entries/<machine_id>-<kernel-version>-<release>.<arch>.conf

Step 6: Manually Rebuild initramfs (If Missing)

# If the initramfs file is not created, manually rebuild it
dracut -f

# Reference: How to rebuild the initial ramdisk image in Red Hat Enterprise Linux

Step 7: Update BLS Configuration (RHEL 8/9)

# If the /boot/loader/entries/<machine_id>-<newversion>.conf file has not updated the initramfs entry
# Reference: How to generate BLS configuration files under /boot/loader/entries in Red Hat Enterprise Linux?

📊 Diagnostic Steps

# 1. Boot from the old kernel or enter rescue mode
# 2. Verify if initrd/initramfs exists in the /boot directory
ls -lh /boot/initramfs-$(uname -r).img

# 3. Verify if initrd declaration exists in the following files:
# RHEL 6 and below: /boot/grub/grub.conf
# RHEL 7 (BIOS): /boot/grub2/grub.cfg
# RHEL 7 (UEFI): /boot/efi/EFI/redhat/grub.cfg
# RHEL 8/9: /boot/loader/entries/<machine_id>-<kernel-version>-<release>.<arch>.conf

# 4. Use df to check if /boot and /tmp partitions are close to 100% full
df -h /boot /tmp

💡 Root Cause

•The initramfs declaration is missing in the boot loader configuration, or the initramfs file itself is missing in the <span><span>/boot</span></span>directory•This may be due to incomplete kernel package installation, which can be caused by:•System hang/crash•Insufficient space in <span><span>/boot</span></span>or <span><span>/tmp</span></span>file systems•Check for third-party modules in initramfs, as sometimes third-party modules can cause such issues

⚠️ 4. Case 4: System Enters Maintenance Mode After Boot

📋 Problem Scenario

After booting, the system enters maintenance mode, displaying the following message:

Give the root password for maintenance 
(Or press Control-D to continue):

Or the system displays an error indicating that the VolumeGroup is not found during boot.

🎯 Solution: 5-Step Repair Process

Step 1: Provide Root Password

# Enter the root password at the prompt
Give the root password for maintenance 
(Or press Control-D to continue):      <-------- Enter root password here

Step 2: Remount Root File System as Read-Write

# Remount the root file system as read-write mode
mount -o remount,rw /

Step 3: Check and Fix /etc/fstab Entries

# Check LVM volume groups and logical volumes
lvm lvs

# Activate all volume groups
lvm vgchange -ay

# Try to mount all fstab entries (verbose mode)
mount -a -v

# View fstab content
cat /etc/fstab

Step 4: Comment Out Failed Mounts

# Based on the output of mount -a -v, identify failed mounts
# Comment out the failed entries in /etc/fstab (add # at the beginning of the line)

# Example:
# /dev/mapper/vg-failed /mnt/failed ext4 defaults 0 0

Step 5: Check and Set Default Target

# Check the system default target
systemctl get-default

# The output should be "multi-user.target" or "graphical.target"
# If not, set the default target
systemctl set-default multi-user.target
# or
systemctl set-default graphical.target

📊 Diagnostic Steps

# In maintenance mode, after providing the root password, execute the following commands:

# 1. Check LVM logical volumes
lvm lvs

# 2. Activate all volume groups
lvm vgchange -ay

# 3. Try to mount all fstab entries
mount -a -v

# 4. View fstab content
cat /etc/fstab

# 5. Compare the output of lvm commands with the output of mount -a -v and the actual entries in /etc/fstab

💡 Root Cause

•<span><span>/etc/fstab</span></span>contains invalid entries•VolumeGroups failing for various reasons•Incorrect target assigned for boot

💡 5. Case 5: File System Corruption (Repair in Rescue Mode)

📋 Problem Scenario

The system cannot boot and needs to repair the file system that the operating system depends on (usually <span><span>/</span></span> or <span><span>/var</span></span>). Or the root file system enters read-only mode.

🎯 Solution: 8-Step Complete Repair Process

Step 1: Boot from Installation Media

# 1. Boot the system from a binary DVD or boot disk
# 2. Use installation media with the same major version as the system
# 3. If possible, download the latest minor version from access.redhat.com
#    For example: Use RHEL 8.10 instead of RHEL 8.0

# RHEL 7/8/9/10: Select Troubleshooting → Rescue a Red Hat Enterprise Linux system
# RHEL 6: Select Rescue installed system
# RHEL 5: At the prompt, enter "linux rescue" (without quotes)

Step 2: Select Language and Keyboard

# Provide language and keyboard information as prompted by the system

Step 3: Disable Network

# When prompted to enable network devices on the system, select: No

Step 4: Skip Mounting

# When prompted, select: Skip

Step 5: Initialize Software RAID (If Used)

# If using software RAID, first initialize the RAID array
mdadm --assemble --scan

Step 6: Activate LVM Volumes (If Used)

# If using LVM, activate volumes for scanning
lvm vgchange -ay
# Output Example:
# 1 logical volume(s) in volume group "VolGroup00" now active

Step 7: Repair File System

For XFS File System:

# Perform a check on the device containing the file system
xfs_repair /dev/mapper/<vg>-<lv>
# or
xfs_repair /dev/<sd device>
# or
xfs_repair /dev/<md device>

# Note: If xfs_repair cannot run, you may need to recreate the log
# This can be done by running xfs_repair -L
# Warning: Ensure you have a known good backup of data on the affected file system before attempting repairs

For EXT File System:

# Perform a check on the device containing the file system
e2fsck -fvy /dev/mapper/<vg>-<lv>
# or
e2fsck -fvy /dev/<sd device>
# or
e2fsck -fvy /dev/<md device>

# Parameter Explanation:
# -f: Force check, even if the file system appears clean
# -v: Verbose mode
# -y: Automatically answer "yes" to all questions

Step 8: Exit Rescue Environment

# Exit the rescue environment and boot the system normally
exit

📊 Video Tutorial

Refer to Red Hat Knowledge Base video: Repairing Filesystems in Rescue Mode. (7:11)

💡 Root Cause

The file system may become corrupted for various reasons, the most significant of which include:

•Connection failure during <span><span>write()</span></span>•Hardware failure (intermittent hardware issues)•Bad cables/fiber•Power outages•Network connection failures•NIC jitter•Software/firmware bugs•Incorrect file system adjustment operations, such as resizing logical volumes

🚨 6. Case 6: System Shuts Down During Boot Process (Insufficient Audit Log Space)

📋 Problem Scenario

The system shuts down before completing the boot process. When attempting to start, the following occurs:

•The system suddenly shuts down during the boot process•No error messages are displayed on the console•The system keeps rebooting

🎯 Solution: Expand /var/log/audit File System or Clean Up Old Audit Logs

⚠️ Important Note:

This issue typically occurs after CIS hardening, so we recommend discussing with your security team before taking any action.

🔧 Diagnostic Steps

Step 1: Add audit=0 Parameter in GRUB Menu

# 1. Boot the system, stop at the GRUB menu
# 2. Press 'e' to edit the boot entry
# 3. Add: audit=0 to the kernel command line
# 4. Press Ctrl+X to continue booting

Step 2: Check Space Used by Audit Logs

# Check the space used by audit logs
grep -e audit -e Mounted df

# Example Output:
# Filesystem              1K-blocks     Used Available Use% Mounted on
# /dev/mapper/rootvg-auditlv01       1038336  987208     51128  96% /var/log/audit

# In the above example, audit logs are stored on a dedicated partition, using 96%

Step 3: Check Audit Configuration

# Check audit configuration
grep ^admin_space_left /etc/audit/auditd.conf

# Example Output:
# admin_space_left = 50
# admin_space_left_action = HALT

# In the above example, the audit configuration is set to shut down the system if space is below 50MB
# From Step 1, the current available space is 51128KB (approximately 50MB), which triggers the shutdown

💡 Root Cause

•The CIS hardening requires that if <span><span>/var/log/audit</span></span>is low on space, the system should shut down to avoid losing potentially useful audit logs for forensic purposes•When this occurs, the system shuts down, but no messages are printed on the console, making troubleshooting very difficult•Bugzilla 2207869 has been submitted on RHEL 9 to enhance this feature

🔧 Solution Options

Option 1: Expand /var/log/audit File System

# 1. If /var/log/audit is on a dedicated partition, expand that partition
# 2. If /var/log/audit is on the root partition, expand the root partition
# 3. Resize the file system
resize2fs /dev/mapper/rootvg-auditlv01  # ext4
xfs_growfs /dev/mapper/rootvg-auditlv01  # xfs

Option 2: Clean Up Old Audit Logs

# 1. Check audit log files
ls -lh /var/log/audit/

# 2. Delete old audit log files (be cautious!)
# It is recommended to back up first
tar -czf /tmp/audit_logs_backup.tar.gz /var/log/audit/

# 3. Delete old logs (e.g., delete logs older than 30 days)
find /var/log/audit/ -type f -mtime +30 -delete

# 4. Or use audit log rotation
# Check max_log_file and num_logs settings in /etc/audit/auditd.conf

Option 3: Adjust Audit Configuration (After Discussion with Security Team)

# 1. Edit audit configuration
vi /etc/audit/auditd.conf

# 2. Adjust admin_space_left value (increase threshold)
# admin_space_left = 100  # Increase from 50MB to 100MB

# 3. Or change admin_space_left_action (discuss with security team)
# admin_space_left_action = SUSPEND  # Suspend instead of shutdown

# 4. Restart audit service
systemctl restart auditd

🎁 Conclusion!

📋 Value Summary

Today, FYC has brought you the complete guide to 6 classic practical cases from Red Hat:

✅ Case 1: Booting from Backup Kernel:

•How to temporarily boot from the old kernel when the new kernel crashes•How to permanently set the default kernel•How to prevent automatic kernel updates

✅ Case 2: Huge Page Configuration Error:

•How to fix huge page configuration errors without using ISO•Handling methods for both kernel command line and configuration file scenarios•Different handling methods for RHEL 7 and RHEL 8/9

✅ Case 3: Missing initramfs:

•7-step repair process for unable to mount root file system after update•How to check /boot and /tmp space•How to manually rebuild initramfs

✅ Case 4: Maintenance Mode:

•5-step repair process for entering maintenance mode after boot•How to check and fix /etc/fstab entries•How to activate LVM volume groups

✅ Case 5: File System Repair:

•Complete process for repairing XFS and EXT file systems in rescue mode•How to initialize RAID and activate LVM•Common causes of file system corruption

✅ Case 6: Insufficient Audit Log Space:

•Diagnostic methods for system shutdown during boot process•How to expand /var/log/audit file system•How to clean up old audit logs

By mastering these 6 classic cases, you will be able to quickly locate and resolve issues when the system fails to boot after an update, ensuring business continuity!

🎯 Call to Action

Do you find this article not enough? Want to see more detailed troubleshooting scripts, boot issue checklists, and more real cases?

👉 Click on my subsequent WeChat mini-program 【FYC’s Knowledge Planet】, to get:

•📚 One-click script for boot failure diagnosis (automated diagnostic process)•🔧 Complete repair scripts for 6 major cases (automated repair for each case)•📊 Boot issue checklist (Checklist)•💡 More analyses of boot failure cases (real production environment scenarios)•🎯 In-depth analysis of boot process (complete process from BIOS to systemd)

FYC’s mission: To make every operations engineer a boot failure diagnosis expert! Technology should be hardcore, and the copy should be engaging! 🔥

#Operations #Linux #Boot Failure #Kernel Crash #File System Repair #Troubleshooting #RCA #Root Cause Analysis #Technical Content #RedHat #RHEL #Official Cases