Communication systems have very high stability requirements, with system availability reaching 99.999% (i.e., interruptions of no more than 5 minutes per year). A significant amount of effort is required to validate stability during testing. This article analyzes an abnormal reboot case on a Linux platform for communication systems, providing methods and operations for fault diagnosis.

Excluding External Power Anomalies

First, it is essential to rule out power supply anomalies, such as power outages, unstable current, or undervoltage that could lead to system abnormalities. This can be determined by whether the system underwent a cold reboot or a hot reboot.

In Linux systems, the main differences between cold reboot and hot reboot are as follows:

• • Reboot Method: A cold reboot is performed by physically cutting off the power and then restoring it, such as directly unplugging the power cord, pressing the power button on the chassis to turn it off and then back on, or using the server’s BMC/IPMI remote command to force a power off and then power on. A hot reboot, on the other hand, is triggered by software commands without turning off the hardware power, such as using <span>reboot</span> or <span>shutdown -r</span> commands.
• • Process Handling: A cold reboot forcibly terminates all processes without giving them a chance to save data. During a hot reboot, the operating system sends signals to all running processes, allowing them to save their current state and shut down properly. However, if the processes do not respond in time or if the system is forcibly rebooted, data loss may occur.
• • Data Preservation: A cold reboot can lead to data in the disk cache not being written to the disk in time due to the sudden power cut, which can easily result in data loss or file system corruption. A hot reboot typically performs disk data synchronization operations (such as fsync), which can normally ensure data is written to the disk, reducing the risk of data loss and file system corruption.
• • Hardware State: A cold reboot clears all hardware caches, including CPU registers and memory data, requiring the hardware to undergo self-checks and other initialization processes again. During a hot reboot, the hardware remains powered on, and a complete hardware self-check is not necessary; only some critical hardware components are re-initialized, resulting in a relatively faster reboot speed.

The type of reboot can be identified from the system logs using the following methods:

• • Check System Log Files: In CentOS/RHEL systems, relevant logs are usually located in <span>/var/log/messages</span>, while in Ubuntu/Debian systems, they can be found in <span>/var/log/syslog</span>. You can use the <span>tail</span> command or a text editor to view the log content, searching for keywords related to reboots, such as “reboot” or “shutdown”.
• • Analyze Log Keywords: If the logs contain records related to “reboot” and there is clear command execution information, such as “reboot: Restarting system”, it usually indicates a hot reboot triggered by a software command. If the logs show power-related keywords, such as “power off” or “power on”, or contain abnormal information like “Kernel panic – not syncing” (kernel crash), and there are no clear software reboot command records, it may indicate a cold reboot, possibly due to power failure or hardware issues.
• • Utilize auditd Logs: For systems with auditd enabled, you can use the <span>ausearch</span> tool to view audit logs. Executing <span>ausearch -k system_shutdown -r recent</span> can show recent shutdown events, while <span>ausearch -k system_boot -r recent</span> can show boot events. If there are two consecutive system boot logs without any shutdown logs in between, it is likely a cold reboot that did not shut down properly.
• • Check last Command Output: The <span>last</span> command displays the history of system logins and reboots. A normal hot reboot will have clear shutdown and reboot record sequences, while a cold reboot may lack normal shutdown records or show abnormal reboot time intervals, which can be used to determine the reboot type in conjunction with specific log content.

Common Analysis Methods

Analyzing with <span>last</span> and <span>lastreboot</span> Commands

<span>last</span> command records the history of system startups, shutdowns, and user logins, with data sourced from the <span>/var/log/wtmp</span> file (which records system session information).

• • Hot Reboot: Under normal circumstances, a hot reboot will first record <span>shutdown</span> or <span>reboot</span> events, followed by the next <span>reboot</span> startup event, with a continuous timeline (for example: <span>reboot system boot 5.4.0-100-generi Thu Jul 24 10:00 still running</span> has a corresponding shutdown record before it).
• • Cold Reboot: Due to sudden power loss, the <span>wtmp</span> file may not have recorded the shutdown event properly, leading to a situation where the startup record directly follows the last startup record, with no shutdown logs in between, or showing <span>crash</span> (crash) related markers.

Example: The system rebooted at 18:05

last | grep reboot  # Filter reboot records
last            # Directly display the most recent reboot information
root@localhost:~# last  
root     pts/0        xx.17.10.xx    Fri Sep  5 18:06   still logged in
reboot   system boot  5.10.xx-xx-dir Fri Sep  5 18:05   still running
root     pts/2        xx.xx.10.xx    Fri Sep  5 18:01 - down   (00:03)
root     pts/1        xx.17.10.xx     Fri Sep  5 10:56 - 18:04  (07:08)

Check File System Consistency and Disk Cache Status

A cold reboot (sudden power loss) may lead to unsynchronized disk cache to the disk, causing file system inconsistencies, while a hot reboot typically triggers <span>sync</span> (synchronize cache to disk) operations.

• • Check the file system check information during startup using the dmesg command:
• • After a cold reboot, the system startup may show <span>fsck</span> (file system check) repair logs, such as: <span>EXT4-fs (sda1): recovery complete</span> (EXT4 file system repair complete), or <span>UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY</span> (manual repair required).
• • After a hot reboot, the file system usually does not require repair, and there will be no such error messages in the startup logs.

Command Example:

dmesg | grep -i "fsck\|recovery\|ext4\|xfs"  # Check file system repair records

Check Hardware Management Tools (e.g., IPMI/BMC Logs)

For servers or out-of-band management devices (such as BMC/IPMI), a cold reboot (power disconnection) will be recorded at the hardware level, while a hot reboot (software-triggered) typically does not trigger hardware power events.

• • Use the IPMI tool ipmitool to view hardware logs:

  ipmitool sel list  # View System Event Log (SEL)

  If the logs show Power Loss, Chassis Power Off followed by Chassis Power On, it confirms a cold reboot; whereas a hot reboot usually does not have such hardware power events, only recording System Reset.

Analyze <span>/proc/uptime</span> and Boot Timestamps

<span>/proc/uptime</span> file records the uptime of the system after booting (the first column is total seconds), which can be combined with <span>/proc/stat</span>‘s <span>btime</span> (the Unix timestamp of system boot) to calculate the boot time.

• • After a cold reboot, if the system had been running for a long time (e.g., several days) but suddenly rebooted without normal shutdown records, combined with hardware logs (such as power events), it can be inferred as an involuntary reboot (cold reboot).
• • A hot reboot is usually planned, and the boot timestamp may match the administrator’s operation time (such as the time when the <span>reboot</span> command was manually executed).

Command Example:

cat /proc/uptime  # Check system uptime
185.97 2269.63

date -d @$(grep btime /proc/stat | awk '{print $2}')  # Convert boot time to readable format
Fri Sep  5 18:09:59 CST 2025

Check Kernel Crash Logs (<span>/var/log/dmesg</span> or <span>/var/crash</span>)

If a cold reboot is due to a kernel crash (e.g., power loss caused by hardware failure), the kernel may log messages to <span>dmesg</span> (the kernel ring buffer in memory) before crashing, or write to the <span>/var/crash</span> directory (if <span>kdump</span> is enabled).

• • A hot reboot typically does not generate kernel crash logs, while a cold reboot accompanied by hardware errors may show messages like <span>Kernel panic - not syncing</span>, <span>Hardware Error</span>, etc., in <span>dmesg</span>.

dmesg | grep -i "panic\|error\|fail"  # Search for kernel error messages
[    2.208313] spi-nor: probe of spi1.1 failed with error -2
[    2.267733] optee: probe of firmware:optee failed with error -22
[    5.733034] EXT4-fs (mmcblk1p4): re-mounted. Opts: errors=remount-ro
[   33.058557] vfio-fsl-mc dprc.2: device_add() failed for device dpdmux.0: -17
ls /var/crash  # Check for crash log files

Conclusion

• • Hot Reboot: Typically has clear software trigger records (<span>reboot</span> command, <span>shutdown</span> logs), file system consistency, no hardware power events, and complete <span>wtmp</span> records.
• • Cold Reboot: Often accompanied by incomplete <span>wtmp</span> records (no shutdown logs), file system repair information, hardware power loss logs (IPMI), or kernel crash information.

By combining multiple methods (logs + hardware records + system status), a more accurate determination of the reboot type can be made.