Cannot Boot CentOS 7 VM After Updating Host CentOS 7 Kernel

Home » CentOS » Cannot Boot CentOS 7 VM After Updating Host CentOS 7 Kernel
CentOS 10 Comments

I have something very strange that occurred. After updating the kernel on my host CentOS 7 Dell 2950iii I have found that one of my CentOS 7
guest VMs will no longer boot… it just stops at the grub prompt (a second VM functions just fine). I have no idea why this problem occurred and have been unable to fix it. On google I have found several suggestions as to how to repair grub but so far none have worked. What I
have found so far is this:

grub> ls
(hd0) (hd0,msdos2) (hd0,msdos1)

grub> ls -l (hd0,1) (hd0,msdos2) (hd0,msdos1)
Partition hd0,1: Filesystem type xfs, UUID
250008ab-9fde-4892-a8b7-a38882b1a1448 – Partition start at 1024KiB –
Total size
512000Kib
Partition hd0,msdos2: No known filesystem detected –
Partition start at
513024KiB – Total size 12069888KiB
Partition hd0,msdos1: Filesystem type xfs, UUID
250008ab-9fde-4892-a8b7-a38882b1a1448 – Partition start at 1024KiB –
Total size
512000Kib

So it appears that (hd0,1) and (hd0,msdos1) are the same partition containing the boot directory contents

grub> set root=(hd0,1)
grub> ls -l /
DIR 20151002231139 grub/
DIR 20161013022100 grub2/
10190975 20151219235512 initrd-plymouth.img
40655493 20150403032637
initramfs-0-rescue-6494b5d98adc4f66b0cf4c19a0f6ab66.img
4902656 20150403032703
vmlinuz0-rescue-6494b5d98adc4f66b0cf4c19a0f6ab66
252739 20161010232025 symvers-3.10.0-327.36.2.el7.x86_64.gz
29666884 20161013012551 initramfs-3.10.0-327.36.2.el7.x86_64.img
2965270 20161010231818 System.map-3.10.0-327.36.2.el7.x86_64
126431 20161010231818 config-3.10.0-327.36.2.el7.x86_64
5157936 20161010231819 vmlinuz-3.10.0-327.36.2.el7.x86_64
18119089 20161013022032
initramfs-3.10.0-327.36.2.el7.x86_64kdump.img

Is that (hd0,msdos2) the root partition? If is, has it become corrupted somehow? Can I get this system to boot somehow so that I might fix the grub configuration? So far I have tried something along these lines with out success:

grub> set root=(hd0,msdos2)
grub> linux (hd0,1)/vmlinuz-3.10.0-327.36.2.el7.x86_64 root=(hd0,msdos2)/
grub> initrd initrd-plymouth.img grub> boot

I just get this error:
[ 2.128131] No filesystem could mount root, tried:
[ 2.129038] Kernel panic – not syncing: VFS: Unable to mount root fs on unknown-block(1,0)
[ 2.130537] CPU: 0 PID: 1 Comm: swapper/0 Not tainted
3.10.0-327.36.2.el7.x86_64 #1
[ 2.132017] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011

{remainder of stack dump}

Does anyone have any ideas as to

1.) How to fix the problem?
2.) Why the problem occurred in the first place?
3.) How to go about debugging the problem?

I would really like to know how this problem could have occurred on the
1 VM but the Host and 2nd Guest are just fine?

Thanks for your help.

10 thoughts on - Cannot Boot CentOS 7 VM After Updating Host CentOS 7 Kernel

  • That is what I thought was strange… there is no matching initrd. The only one there was the one I tried. I checked all the working systems both hosts and guests and they all just have a initrd-plymouth.img.

  • Nothing obvious. I checked /var/log/messages, /var/log/libvirt/qemu,
    /var/log/libvirt/lxc & /var/log/qemu-ga. The /var/log/libvirt/lxc &
    /var/log/qemu-ga were empty. The /var/log/libvirt/qemu directory had a log file of interest Outgoing-CentOS-7-VM.log but nothing in it that tells me anything obvious.

    2016-10-30 06:40:09.764+0000: shutting down
    2016-10-30 06:40:16.926+0000: starting up libvirt version: 1.2.17, package: 13.el7_2.5 (CentOS BuildSystem <http://bugs.CentOS.org>,
    2016-06-23-14:23:27, worker1.bsys.CentOS.org), qemu version: 1.5.3
    (qemu-kvm-1.5.3-105.el7_2.7)
    LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=spice /usr/libexec/qemu-kvm -name Outgoing-CentOS-7-VM -S
    -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -cpu Penryn -m 4096
    -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
    6494b5d9-8adc-4f66-b0cf-4c19a0f6ab66 -no-user-config -nodefaults
    -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-Outgoing-CentOS-7-VM/monitor.sock,server,nowait
    -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet
    -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1
    -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7
    -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5
    -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1
    -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2
    -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/vm-images/CentOS7.0.qcow2,if=none,id=drive-virtio-disk0,format=qcow2
    -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1
    -drive if=none,id=drive-ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd%,id=hostnet0,vhost=on,vhostfd’ -device virtio-net-pci,netdev=hostnet0,id=net0,macR:54:00:7b:a5:c2,bus=pci.0,addr=0x3
    -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev spicevmc,id=charchannel0,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
    -device usb-tablet,id=input0 -spice portY00,addr7.0.0.1,disable-ticketing,seamless-migration=on -vga qxl -global qxl-vga.ram_sizeg108864 -global qxl-vga.vram_sizeg108864
    -global qxl-vga.vgamem_mb -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -device virtio-balloon-pci,id

  • So I booted off the CentOS-7-x86_64-DVD-1511.iso and everything looks just fine:

    > df Filesystem 1K-blocks Used Available
    Use% Mounted on
    /dev/mapper/live-rw 2030899 949022 1077781 47% /
    devtmpfs 2004040 0 2004040
    0% /dev tmpfs 2023652 0
    2023652 0% /dev/shm tmpfs 2023652 8520 2015132
    1% /run tmpfs 2023652 0
    2023652 0% /sys/fs/cgroup
    /dev/sr1 4227724 4227724 0 100%
    /run/install/repo tmpfs 2023652 200 2023452
    1% /tmp
    /dev/mapper/CentOS-root 10799104 3894196 6904908 37% /mnt/sysimage
    /dev/vda1 508588 143516 365072 29%
    /mnt/sysimage/boot tmpfs 2023652 0
    2023652 0% /mnt/sysimage/dev/shm

    > ls /mnt/sysimage bin boot dev etc home lib lib64 media misc mnt net
    opt proc root run sbin srv sys tmp usr var

    > ls -l /mnt/sysimage/boot total 109424
    -rw-r–r–. 1 root root 126431 Oct 10 23:18
    config-3.10.0-327.36.2.el7.x86_64
    drwxr-xr-x. 2 root root 26 Oct 2 2015 grub drwx——. 6 root root 104 Oct 13 02:21 grub2
    -rw-r–r–. 1 root root 40655493 Apr 3 2015
    initramfs-0-rescue-6494b5d98adc4f66b0cf4c19a0f6ab66.img
    -rw——-. 1 root root 29666884 Oct 13 01:25
    initramfs-3.10.0-327.36.2.el7.x86_64.img
    -rw——-. 1 root root 18119089 Oct 13 02:20
    initramfs-3.10.0-327.36.2.el7.x86_64kdump.img
    -rw-r–r–. 1 root root 10190975 Dec 19 2015 initrd-plymouth.img
    -rw-r–r–. 1 root root 252739 Oct 10 23:20
    symvers-3.10.0-327.36.2.el7.x86_64.gz
    -rw——-. 1 root root 2965270 Oct 10 23:18
    System.map-3.10.0-327.36.2.el7.x86_64
    -rwxr-xr-x. 1 root root 4902656 Apr 3 2015
    vmlinuz0-rescue-6494b5d98adc4f66b0cf4c19a0f6ab66
    -rwxr-xr-x. 1 root root 5157936 Oct 10 23:18
    vmlinuz-3.10.0-327.36.2.el7.x86_64

    So the CentOS DVD iso in linux rescue mode shows that everything is there and can be mounted. I guess that means somehow either grub itself is corrupted or one of the boot images. So is there a way for me to generate a new initrd while booted in linux resuce mode or will re-installing grub help? How would I attempt re-installing grub while booted in linux rescue mode?

  • Thank you for the hint. The way I fixed this problem was to do this:

    1.) Booted into linux rescue mode using the CentOS-7-x86_64-DVD-1511.iso
    2.) chroot /mnt/sysimage
    3.) Used nmtui to setup a network
    4.) yum update (installed the 3.10.0-327-36.3.el7.x86_64 kernel)
    5.) grub2-mkconfig -o /boot/grub2/grub.cfg 6.) yum reinstall kernel*-3.10.0-327-36.3.el7.x86_64

    Step 5 was necessary because of a grubby error indicating that there was no suitable template. Step 6 was necessary because according to the googled page I found regarding step 5 indicated that the default kernel would not be that found in step 4. I probably could have skipped step 4
    and just done steps 5 and 6. But the VM is now up and running the latest kernel.

    Now the question is how did this happen. My best guess is that the VM
    was unhappy when I booted the host while it was running. Is rebooting the host without shutting down guests first a risky thing to do?

  • I’ve seen something similar when installing a kernel if /etc/fstab didn’t match df. Mkinitrd bombs out leaving the system unbootable. The rescue .iso/mkinitrd path you followed was the fastest way to get the system up.