Click the blue text to follow us // A journey of a thousand miles begins with a single step
【Problem Description】
The virtual machine fails to boot and shuts down immediately after starting.
Log in to the host machine corresponding to the problematic virtual machine and attempt to start the virtual machine.
virsh start problematic_vm_UUIDAfter executing the start command, immediately attempt to log in to the console.
virsh console problematic_vm_UUIDThe console output log reports an error: systemd-shutdown[xxx] Failed to remount ‘/’ read-only: Device or resource busy.
systemd-shutdown[1]: Failed to wait for process: Protocol error.
reboot: Power down.
【Problem Analysis】
Preliminary judgment: This error is generally caused by the system disk being full.
【Solution】
1. Identify the UUID of the problematic cloud host and locate the corresponding host machine.
Log in to the host machine corresponding to the problematic cloud host and create a mount directory.
mkdir -v /tmp/test2. Log in to the host machine corresponding to the problematic cloud host and check the cloud host’s configuration file. Find the problematic virtual machine’s UUID.xml file in /etc/vm/ or /etc/kvm/vm and view the problematic cloud host’s UUID.xml file.
cat problematic_vm_UUID.xml3. Locate the system disk vsimage of the problematic virtual machine, find the details of the system disk, and in the device section, find the file path of the virtual machine’s system disk.
The corresponding disk identifier is usually: vda, mounted in the system at /dev/vda1.
4. Log in to the host machine corresponding to the problematic virtual machine and map the system disk of the problematic virtual machine to the network device nbd3 at the host level.
Before performing the operation, first shut down the problematic virtual machine to prevent data loss during the boot operation, and do not perform any boot-related operations on the shut-down virtual machine during the mounting process.
Map the virtual machine’s system disk file to the host’s local device /dev/nbd3. A device /dev/nbd3p1 will appear under /dev; if the system disk has multiple partitions, multiple /dev/nbd3p* files may appear.
Before mapping: ll /dev/nbd3 Mapping operation: qemu-nbd -c /dev/nbd3 /dev/disk/by-id/scsi-1TENCENT_aaaaa1a-bbbbe53-6666-ccccc3d576 After mapping: lsblk | grep nbd35. Log in to the host machine corresponding to the problematic virtual machine and perform a temporary mount, using the system disk of the problematic virtual machine as a temporary data disk mounted to the previously created /tmp/test directory.
/dev/nbd3p1 mounted to the local directory (make sure to create /tmp/test first).
mount /dev/nbd3p1 /tmp/test lsblk | grep nbd36. Check the system disk usage and view the virtual machine’s system disk files in the mount path.
df -hcd /tmp/testClear the corresponding var/log directory messagexxx files and audit log files using echo > to free up space.
Check if the /etc/fstab file configuration is correct.
7. After processing, it is necessary to unmount the device and disconnect the nbd connection.
umount /tmp/test/Cancel the mapping of the nbd3 device corresponding to the problematic virtual machine on the host machine.
qemu-nbd -d /dev/nbd38. Start the problematic virtual machine, it should boot normally, log in to the system to check recovery.
virsh start problematic_vm_UUID
Final Thoughts
Preparation leads to success; lack of preparation leads to failure. Planning ahead prevents confusion, and setting things in order beforehand avoids difficulties. A healthy body and the tranquility and joy that come with it, along with an active, clear, and profound understanding of things, and a gentle, moderate desire that results in a clear conscience—these benefits cannot be replaced by wealth or status. (Arthur Schopenhauer)
