Issues Booting CentOS7 [dracut Is Failing To Enable CentOS/root, CentOS/swap LVs]

Home » CentOS » Issues Booting CentOS7 [dracut Is Failing To Enable CentOS/root, CentOS/swap LVs]
CentOS 10 Comments

Hello,

I’m having a strange problem booting a new CentOS7 installation. Below some background on this. [I have attached the tech details at the bottom of this message]

I started a new CentOS7 installation on a VM, so far all good, o/s boots fine. Then I decided to increase VM disk size (initially was 10G) to 13G. Powered off the VM, increased the vhd via the hypervisor, booted from CentOS livecd, selected “recover my CentOS installation”. Then I used the following sequence of commands to make the new vhd size “visible” to the o/s …

– ran fdisk /dev/sda and deleted partition 2. (This is the LVM partition where the o/s is stored. The first partition is /boot [xfs] partition *non lvm*.)
– created a new partition with same starting sector as before deleting it, but a different ending sector (to reflect to the increased space).
– set partition type to 8e (lvm), saved settings and exit.
– ran pvresize and lvresize to make new space visible to the o/s.All good, I can see the space increase on ‘lvdisplay CentOS/root’.
– ran ‘mount /dev/mapper/CentOS-root /mnt/root’ to temporarily mount the o/s partition.
– ran ‘xfs_growfs /mnt/root’ to resize the o/s fs to the new size. It was successful and I could actually chroot to the o/s and verify new disk size.
– rebooted and tried to boot from hdd this time. Grub menu shows up and loads default kernel (3.10.0-862.9.1.el7.x86_64).
– after initial kernel boot up process, booting continues to initrd and then it’s where the problem starts…
– looks like dracut has issues locating/enabling /dev/mapper/CentOS-root
(lv) and as a result it cannot boot to the ‘real’ root fs (/).
– while in dracut shell, I execute the following command sequence:
1. lvm lvchange -ay /dev/CentOS/root
2. lvm lvchange -ay /dev/CentOS/swap
3. ln -s /dev/mapper/CentOS-root /dev/root
4. exit
…and the o/s boots fine…so looks like the pv,vg,lv is detected properly while in initrd, but somehow dracut has difficulties enabling the root,swap LVs ?

– While in o/s, I rebuild initrd by using: ‘dracut -f -v -a lvm’
Note, I have to use ‘-a lvm’ as for some reason, if don’t, the lvm utils
(lvm,lvm_scan) are not being included to the initrd, not sure why this happens. After rebooting, same thing happens, I have to manually boot the system via dracut shell.

I’m a bit stuck at this point, any clues what I’m doing wrong in here ?

As I previously said, this is a new installation, so I could simply reinstall the whole thing, but I would rather try to find out some answer to why is this happening … :-) [curiosity]

Some additional details:
==================
I’ve attached /run/initramfs/rdsosreport.txt file [generated while in inird] here:
https://privatebin.net/?fdc4052c0c402884#gdB/QYR3IeR55SxUbjfrkZPQfJ7jMxiUxq5DWhTVbWc=

o/s ver=CentOS Linux release 7.5.1804 (Core)

[root@localhost ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 13G 0 disk
├─sda1 8:1 0 500M 0 part /boot
└─sda2 8:2 0 12.5G 0 part
├─CentOS-root 253:0 0 11.5G 0 lvm /
└─CentOS-swap 253:1 0 1G 0 lvm [SWAP]

[root@localhost ~]# cat /etc/default/grub GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=”$(sed ‘s, release .*$,,g’ /etc/system-release)”
GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT=”console”
GRUB_CMDLINE_LINUX=”rd.lvm.lv=CentOS/root rd.lvm.lv=CentOS/swap rd.shell rhgb”
GRUB_DISABLE_RECOVERY=”true”

10 thoughts on - Issues Booting CentOS7 [dracut Is Failing To Enable CentOS/root, CentOS/swap LVs]

  • Hi,

    At this point I usually reboot and do the following steps while the system is running. I don’t say how you do it is wrong, it’s just easier/faster to do things while the system is online if downtime matters.

    I don’t really know what causes your issues. The reason may be that I have more and more problems understanding the whole boot process and all the variants we used to have over more than two decades…

    But, what happens if you let the kernel post install scripts do the work or setting up the initrd things? If you reinstall the latest kernel, does it change anything?

    Regards, Simon

  • Initially, I just rebooted from LiveCD and left Grub,kernel and dracut do their job with the defaults, but unfortunately, boot process stops at initrd (dracut) stage with the following error:

    [ 197.768159] localhost.localdomain dracut-initqueue[252]: Warning: Could

    Then, I’m dropped to dracut shell, where I boot the system by using the lvm commands as described to my first post. Once in O/S, I re-create initrd by using ‘dracut -f -v -a lvm’ command and reboot again the system. But still I get the same error during boot …

    If you reinstall the latest kernel, does

    This is actually the latest kernel (3.10.0-862.9.1.el7.x86_64) but I tried to boot with the second (older) kernel
    (vmlinuz-3.10.0-327.36.3.el7.x86_64), where I got exactly the same error during boot.

    Clearly, this is not a kernel issue, perhaps not even a dracut (initrd)
    issue, since everything was working fine, before resizing the PV/LV. I
    suspect that something went wrong during that step, but what? As I can successfully boot the system manually via dracut, normally everything should be ok? no?

    What can prevent dracut from activating the 2 LVs (CentOS/root and CentOS/swap) during initial boot phase ?

    Thanks, Yannis

  • Suggestion: once it’s up, rebuild the initramfs. I tried that already, but still the same problem. Aparently dracut does not want to activate the LVs required to boot to the root filesystem, for some reason …

    Yannis

  • Yannis Milios wrote:
    At this point, I’d start wondering about the grub2 defaults, and the kernel command line.

    mark

  • On Monday, July 30, 2018, mark
    This is Grub config..

    [root@localhost ~]# cat /etc/default/grub GRUB_TIMEOUT=5
    GRUB_DISTRIBUTOR=”$(sed ‘s, release .*$,,g’ /etc/system-release)”
    GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT=”console”
    GRUB_CMDLINE_LINUX=”rd.lvm.lv

  • And when you did so, did you remove the “rd.shell” from GRUB_CMDLINE_LINUX
    before?

    Regards, Simon

  • And when you did so, did you remove the “rd.shell” from GRUB_CMDLINE_LINUX
    Yes, basically that option was not there by default, I added it during the troubleshootig steps. Normally, rd.lvm.lv

  • Finally, I was able to find a solution for this. I’m posting the result here, just in case that it’s useful for someone else in the future …

    The problem was not in initramfs (dracut), but on the fact that /dev/sda2
    partition(PV), after I resized it, somehow got 2 partition table signatures.

    One was set as a “dos” partition with an offset “0x1fe” and second set as
    “LVM_member” with an offset “0x8e”.

    This confused blkid in initramfs, during the initial boot stage, thinking that /dev/sda2 is *not* an LVM_member but rather a simple “dos” partition, hence refused to activate CentOS/root and CentOS/swap LVs which were required to boot the machine to the O/S.

    What’s interesting is that neither fdisk or parted were showing this second signature, however there was a trace of it in the dracut report file
    (/run/initramfs/rdsosreport.txt), specifically the one below..

    + blkid
    /dev/sr0: UUID=”2016-10-28-12-18-36-00″ LABEL=”CentOS 7 x86_64″
    TYPE=”iso9660″ PTTYPE=”dos”
    /dev/sda1: UUID=”a620a180-3a8c-4b5f-ad30-804f131a7261″ TYPE=”xfs”
    /dev/sda2: PTTYPE=”dos”
    + blkid -o udev ID_FS_UUID 16-10-28-12-18-36-00
    ID_FS_UUID_ENC 16-10-28-12-18-36-00
    ID_FS_LABEL

  • Yannis Milios wrote:

    Glad you solved it. Btw, in the future, if you have a problem that you finally get resolved, after much difficulty, and go to post the solution, add [SOLVED] to the subject line. Gets people’s attention.

    mark