💾 Linux Storage Failure Emergency Guide | File System Crashed? LVM Error? These 5 Tips Will “Resurrect” Your Data!

📖 Introduction: Do You Feel This Way Too?

First Paragraph: Pain Point Scenario

Have you ever encountered this nightmare scenario? The system suddenly crashes, and after rebooting, it prompts “file system corrupted,” making all data inaccessible. You desperately try to fix it, but each time you get “mount failed” or “superblock error.” Even worse, your boss calls: “Can the data in the database be recovered? Can our business continue?”

Or, you just expanded the LVM, and the system reports “Volume group not found,” making all logical volumes disappear. The user’s data is still there, but you are at a loss…

If you have also experienced such moments of “storage equals despair,” then today FYC has good news for you:Storage failures actually follow a pattern; once you master the correct methods, you can make data “come back to life” like a data recovery expert!

Moreover, we will also discuss thecomplete architecture of the Linux storage stack. Only by understanding how data flows from applications to disks can you quickly locate the problem when a failure occurs.

Second Paragraph: Overview of Solutions

Today we will talk aboutLinux storage failure troubleshooting, a survival skill that every operations engineer must master. We will start with the complete architecture of the Linux storage stack and guide you step by step on how to recover a damaged file system, fix LVM configuration errors, recover lost encrypted volumes, and quickly resolve iSCSI storage network issues.

This article will bring you:

•🎯 Panorama of the Linux Storage Stack: A detailed explanation of the 10-layer architecture from VFS to physical hardware•🔧 File System Recovery Tools: XFS/ext4 damage repair, superblock recovery•📊 LVM Troubleshooting: vgcfgrestore to recover LVM configuration, detailed explanation of archiving features•🔐 LUKS Encrypted Volume Recovery: Header backup and recovery, decryption troubleshooting•🌐 iSCSI Problem Diagnosis: Network connection troubleshooting, authentication configuration, login failure resolution•💡 Practical Case Analysis: The complete process of storage failure troubleshooting in real scenarios

Follow FYC, and say goodbye to the era of “storage equals despair”!

Third Paragraph: 5-Dimension Rating Table

Dimension	Rating	Description
Difficulty Level	⭐⭐⭐⭐⭐	Requires in-depth understanding of storage stack architecture and data recovery principles
Practical Value	⭐⭐⭐⭐⭐	Storage failures often lead to data loss, making it the most urgent type of failure
Technical Depth	⭐⭐⭐⭐⭐	Comprehensive coverage from storage stack architecture to data recovery
Operability	⭐⭐⭐⭐⭐	All commands and tools can be used directly
Urgency	⭐⭐⭐⭐⭐	Storage failures are usually the most urgent and severe type of failure

📚 Main Content: Packed with Useful Information, But Needs to Be “Fed to You”!

🏗️ 1. Panorama of the Linux Storage Stack: Understanding the Complete Path of Data Flow

Before diving into fixing storage failures, FYC wants to draw you a “map.” Only by understanding how data flows from applications to disks can you quickly locate the problem when a failure occurs.

📋 10-Layer Architecture of the Storage Stack: From Application to the Bottom Layer

The Linux storage stack is divided into10 layers, the complete path of data flowing from applications to physical disks:

Application Layer
    ↓
Virtual File System (VFS)       # Layer 1: Unified interface, cache management
    ↓
File System Layer (XFS/ext4)    # Layer 2: File organization, metadata management
    ↓
Device Mapper                  # Layer 3: LVM, LUKS, multipath
    ↓
Block Layer                    # Layer 4: I/O scheduling, request merging
    ↓
DM Multipath                  # Layer 5: Multipath aggregation
    ↓
SCSI Mid-layer                # Layer 6: SCSI protocol bridging
    ↓
Low-level Drivers             # Layer 7: Hardware drivers
    ↓
Transport Layer               # Layer 8: SATA/SAS/FC/iSCSI
    ↓
Host Bus Adapter              # Layer 9: Physical interface card
    ↓
Physical Hardware             # Layer 10: Hard disk/SSD/storage array

💡 Key Insight: If any layer among these 10 layers has a problem, data will be inaccessible! Most storage failures occur inLayers 2-4, which are the file system, device mapper, and block layer.

🔍 Detailed Explanation of Each Layer: Functions and Tools

Layer 1: Virtual File System (VFS) 🗂️

VFS provides a unified POSIX interface for applications, regardless of the underlying file system, applications use the same system calls:

# System calls provided by VFS
open(), read(), write(), close(), mmap()...

# Caches maintained by VFS (to improve performance)
- inode cache: file metadata cache
- dentry cache: directory entry cache
- page cache: file data cache (most important!)

# Bypass page cache (commonly used in databases)
Use O_DIRECT flag to open the file

💡 Tip: If you suspect a cache issue, you can try:

# Clear page cache (dangerous operation, only in testing environment!)
echo 1 > /proc/sys/vm/drop_caches

# View cache statistics
free -h
cat /proc/meminfo | grep -i cache

Layer 2: File System Layer (Filesystems) 📁

The file system is responsible for organizing and naming files:

# Default file system in RHEL 7
XFS      # Default, suitable for large files
ext4     # Default in RHEL 6, good compatibility
ext3     # Deprecated
ext2     # Deprecated

# Types of file systems
- Block storage: XFS, ext4 (local disks)
- Network storage: NFS, SMB (network file systems)
- Pseudo file systems: procfs, sysfs (virtual file systems)
- Memory file systems: tmpfs (memory supported)

Layer 3: Device Mapper 🔧

Device Mapper maps logical devices to physical devices, LVM and LUKS rely on it:

# View device mapping
dmsetup ls
dmsetup table

# Example: LVM logical volume mapping
/dev/mapper/myvg-mylv → /dev/dm-0 → /dev/vdb1 + /dev/vdb2

Layer 4: Block Layer ⚡

The block layer is responsible for I/O scheduling and request optimization:

# Three I/O schedulers in RHEL 7
1. deadline   # Default, favors read operations, suitable for most scenarios
2. cfq        # Completely Fair Queuing, default in RHEL 6, suitable for SATA
3. noop       # First In First Out, suitable for devices that do their own scheduling (storage arrays)

# View scheduler
cat /sys/block/sda/queue/scheduler

# Modify scheduler
echo deadline > /sys/block/sda/queue/scheduler

# blk-mq new mechanism (RHEL 7.1+, used by virtio devices)
# Bypass traditional scheduler, use multi-queue mechanism

Layers 5-7: Multipath, SCSI Mid-layer, Low-level Drivers 🔌

These layers handle hardware interfaces and protocol conversions:

# Multipath aggregates multiple I/O paths
multipath -ll

# Identify SCSI devices
lsscsi

# View device information
lsblk

🔧 2. Recovering from File System Damage: Bringing Files “Back to Life”

File system damaged after a system crash? Don’t panic! FYC will teach you how to fix it step by step.

📊 File System Selection: XFS vs ext4

RHEL 7 defaults to using XFS, no longer ext4:

Feature	XFS	ext4
RHEL Version	Default in RHEL 7	Default in RHEL 6
Large File Support	Very good (max 8EB)	Good (max 16TB)
Logging Feature	Yes	Yes
Repair Tool	xfs_repair	e2fsck
Repair Speed	Fast	Slower

⚠️ Important Reminders Before Repair: Three Things Must Be Done

Before proceeding with repairs, FYC wants to remind you of three things:

1First resolve hardware issues: If the damage is caused by hardware failure, fix the hardware first!2Check before repairing: Before repairing, use the <span><span>-n</span></span> option to check, simulating the run3Must unmount first: Repairing on a mounted file system will cause further damage!

# ❌ Incorrect Method: Repair directly on the mounted file system
xfs_repair /dev/vdb1  # This will damage the file system!

# ✅ Correct Method: Unmount first
umount /dev/vdb1
xfs_repair /dev/vdb1

🔍 Check the File System: Discover Problems to Fix Them

Check ext3/ext4 file systems:

# Install tools
yum install -y e2fsprogs

# Check the file system (do not repair, just check)
e2fsck -n /dev/vdb1

# Exit code meanings:
# 0: No errors
# 4: File system errors not corrected
# 8: Operational error
# 16: Usage or syntax error

# Detailed mode check
e2fsck -nv /dev/vdb1

Check XFS file systems:

# Install tools
yum install -y xfsprogs

# Check XFS file system (must unmount!)
umount /dev/vdb1
xfs_check /dev/vdb1    # Old version command
xfs_repair -n /dev/vdb1  # New version check (do not repair)

# xfs_repair -n only checks, does not repair

🛠️ Repair the File System: Let Data See the Light Again

Repair ext3/ext4 file systems:

# 1. Unmount first (important!)
umount /dev/vdb1

# 2. Check the file system
e2fsck -n /dev/vdb1

# 3. Repair the file system
e2fsck -p /dev/vdb1    # -p: auto repair
# or
e2fsck -y /dev/vdb1    # -y: answer "yes" to all issues

# 4. Detailed mode repair
e2fsck -vy /dev/vdb1

# 5. Use backup superblock for repair (if the main superblock is damaged)
e2fsck -b 32768 /dev/vdb1  # 32768 is the backup superblock location

Repair XFS file systems:

# 1. Unmount first (important!)
umount /dev/vdb1

# 2. Check the file system
xfs_repair -n /dev/vdb1

# 3. Repair the file system
xfs_repair /dev/vdb1

# 4. If the log is damaged, zero the log (dangerous!)
xfs_repair -L /dev/vdb1
# ⚠️ Warning: -L option will zero the log, which may cause data inconsistency!

# 5. Remount to verify
mount /dev/vdb1 /mnt
ls -l /mnt

Practical Case: Recovering a Damaged XFS File System

# Scenario: After a system crash, the XFS file system on /dev/vdb1 cannot be mounted

# Step 1: Check the file system status
xfs_repair -n /dev/vdb1
# Output: Found inconsistency issues

# Step 2: Try to mount (confirm cannot mount)
mount /dev/vdb1 /mnt
# Error: mount: wrong fs type...

# Step 3: Unmount (if already mounted)
umount /dev/vdb1

# Step 4: Repair the file system
xfs_repair /dev/vdb1

# Step 5: If repair fails, try to zero the log (cautiously!)
xfs_repair -L /dev/vdb1

# Step 6: Remount
mount /dev/vdb1 /mnt/etc_restore

# Step 7: Check files
ls -l /mnt/etc_restore

# Step 8: Recover orphaned files (if there is a backup)
tar -xzf /root/etc.tgz -C /mnt/etc_restore --keep-newer-files

Case 2: LVM Configuration Recovery

Scenario: After expansion, <span><span>/mnt/lvm</span></span> cannot be accessed, LVM shows abnormal.

Steps to Troubleshoot:

# Step 1: Check LVM status
vgs
lvs
# Found: Volume group or logical volume shows abnormal

# Step 2: Check system logs
journalctl -u lvm2-monitor.service | tail -20

# Step 3: Check LVM archives
vgcfgrestore -l myvg

# Step 4: View recent archive files
ls -lt /etc/lvm/archive/ | head -5

# Step 5: Deactivate logical volume (if activated)
lvchange -an /dev/myvg/mylv

# Step 6: Restore from archive
vgcfgrestore -f /etc/lvm/archive/myvg_20240101.vg myvg

# Step 7: Reactivate
lvchange -ay /dev/myvg/mylv

# Step 8: Remount
mount /dev/myvg/mylv /mnt/lvm

# Step 9: Verify data
ls -l /mnt/lvm
df -h /mnt/lvm

Case 3: Recovering LUKS Encrypted Volume

Scenario: After changing the LUKS password, the new password does not work and needs recovery.

Steps to Troubleshoot:

# Step 1: Check LUKS information
cryptsetup luksDump /dev/vdb2

# Step 2: Try using the old password
cryptsetup luksOpen /dev/vdb2 secure
# Enter old password: RedHatR0cks!

# Step 3: If the old password also does not work, check if there is a header backup
ls -lh /root/luks_header_backup

# Step 4: Test the backup header file (important!)
cryptsetup luksOpen /dev/vdb2 test_secure --header /root/luks_header_backup
# If successful, it means the backup is correct

# Step 5: Close the test
cryptsetup close test_secure

# Step 6: Restore the header file
cryptsetup luksHeaderRestore /dev/vdb2 --header-backup-file /root/luks_header_backup

# Step 7: Use the old password to decrypt
cryptsetup luksOpen /dev/vdb2 secure
# Enter password: RedHatR0cks!

# Step 8: Remount
mount /dev/mapper/secure /mnt/secure

# Step 9: Verify data
ls -l /mnt/secure

Case 4: Comprehensive iSCSI Troubleshooting

Scenario: Cannot access the encrypted volume on iSCSI storage, need to troubleshoot network, iSCSI, and LUKS issues.

Steps to Troubleshoot:

# Step 1: Check iSCSI connection
iscsiadm -m session
# If empty, it means not connected

# Step 2: Check network connection
ping target_server_ip
nc -v target_server_ip 3260

# Step 3: Discover targets
iscsiadm -m discovery -t sendtargets -p target_server_ip

# Step 4: Log in to the target
iscsiadm -m node -T iqn.2016-01.com.example.lab:iscsistorage --login

# Step 5: Verify device
lsblk
# Should see the new iSCSI device, e.g., /dev/sdc

# Step 6: Check LVM volume group
vgs
# Should see the volume group on the iSCSI device

# Step 7: Activate logical volume
lvchange -ay /dev/save/old

# Step 8: Mount logical volume (including LUKS header backup)
mount /dev/save/old /mnt/save

# Step 9: Restore LUKS header
cryptsetup luksHeaderRestore /dev/sdc1 --header-backup-file /mnt/save/luks/iscsistorage_luks_header

# Step 10: Decrypt with password
cryptsetup luksOpen /dev/sdc1 finance
# Enter password: RedHatR0cks!

# Step 11: Mount encrypted volume
mount /dev/mapper/finance /mnt/finance

# Step 12: Verify
ls -l /mnt/finance

🎁 Conclusion!

📋 Value Summary

Today FYC brought you the complete guide to Linux storage failure troubleshooting:

✅ Panorama of the Linux Storage Stack:

•Detailed explanation of the 10-layer architecture from VFS to physical hardware•Understanding the data flow path, quickly locating the problem

✅ File System Recovery Tools:

•XFS/ext4 damage repair, superblock recovery•Complete process of check → repair → verify

✅ LVM Troubleshooting:

•vgcfgrestore to recover LVM configuration•LVM archiving feature: a lifesaver in critical moments

✅ LUKS Encrypted Volume Recovery:

•Header backup and recovery, decryption troubleshooting•Solutions for forgotten passwords

✅ iSCSI Problem Diagnosis:

•Network connection troubleshooting, authentication configuration•Complete process of discovery → login → usage

Mastering these skills will allow you to “resurrect” data in critical moments, upgrading from a “data rescue team member” to a “storage recovery expert”!

🎯 Call to Action

Do you find this article not enough? Want to see more detailed Linux storage stack architecture diagrams, in-depth analysis of file system repairs, and more practical case studies of storage failure troubleshooting?

👉 Click the “Read Original” below to get:

•📚 Complete Storage Failure Troubleshooting Checklist (Checklist)•🔧 Detailed Diagram of the 10-Layer Architecture of the Linux Storage Stack (Visual)•📊 Quick Reference Table for File System Repair Tools (All commands and parameters)•🎯 Best Practices for LVM Archiving Management (Backup and recovery process)•💡 More Real Storage Failure Case Analyses (Covering XFS/LVM/LUKS/iSCSI)•🔐 LUKS Header Backup Script (Automated backup tool)

FYC’s Mission: To enable every operations engineer to become a storage recovery expert! Technology must be hardcore, and the copy must be engaging!🔥

#Operations #Linux #StorageFailure #FileSystem #LVM #LUKS #iSCSI #DataRecovery #TechnicalContent #RedHat #RCA