8.2.2004 Latest Yum Update Renders Machine Unbootable

Home » CentOS » 8.2.2004 Latest Yum Update Renders Machine Unbootable
CentOS 44 Comments

I am running an Intel x64 machine using UEFI to boot an SSD.

Installing the latest yum update which includes grub2 and kernel
4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable, blank screen where grub should be, no error messages, just hangs.

After some hours I managed to modify another bootable partition
(containing older software) and boot it from there.

After that, I  found out it is a known problem.

The main point of this message is to make people aware of the problem and suggest admins don’t run ‘yum update’ until they understand the problem and have a fix at hand.

See ‘UEFI boot blank screen post update’ for a solution and directions to the redhat article.

Regards

Alan

44 thoughts on - 8.2.2004 Latest Yum Update Renders Machine Unbootable

  • Il 31/07/20 13:08, ja ha scritto:
    Me too. Luckily it happened on a test machine.

    Sorry but seems that those packages were not tested before pushing them in the update repo. Would be great to know what happened to the mainstream chains and how a package like grub reached the update repo when it has serious problem (genuine curiosity but not to blame them).

    In my case, the restore procedure reported by RH in case of reboot does not work and reports that the packages are already at the lowest version and that the downgrade is not possibile. I don’t know why.

  • I would add,

    RedHat is recomending also to blacklist grub, shim and mokutil in yum.conf until the problem is resolved.

    Il 31/07/20 18:26, Laack, Andrea P ha scritto:

  • Of course it was tested before it was pushed. Obviously this is not a problem with every install. Surely you don’t think we push items without doing any testing. Certainly not items as important as this update.

    The CentOS infrastructure has hundreds of servers, most of them were not impacted (as an example). In fact, we seem to have had this happen on only one machine in those hundreds so far. It is a problem, obviously.

    The issue seems to be with the shim package (not the grub or kernel packages) and we are currently working with Red Hat on a fix. This issue happened in many Linux OSes and even Windows, not just RHEL and CentOS.

    We will push a fix as soon as one is available.

    I would hold off on installing this until we release the new fixes.

  • It tanked my machine. I managed to get it to boot into emergency mode where I eventually got it to boot on an old kernel as shown in the paste below. After getting it to boot I used grubby to set the default kernel to the oldest one on the machine. Surprisingly enough neither of the two most recent kernels would boot even though the machine was running on the second oldest kernel when I ran the update that tanked it. That adds credence to the comment above that the problem was not the new kernel since it trashed both the new kernel and the one before. Both were from the same series. The 147 kernel works but neither of newer ones would.

    CentOS Linux release 8.2.2004 (Core)

    enp5s0 IP Address = 192.168.15.131

    Linux localhost.localdomain 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9
    13:49:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

    Number of cores = 32
    00:55:13 up 8:37, 1 user, load average: 0.31, 0.48, 0.44

    amdgpu-pci-0900
    Adapter: PCI adapter vddgfx: +0.83 V
    fan1: 771 RPM (min = 0 RPM, max = 3700 RPM)
    temp1: +45.0°C (crit = +94.0°C, hyst = -273.1°C)
    power1: 31.07 W (cap = 125.00 W)

    acpitz-virtual-0
    Adapter: Virtual device temp1: +16.8°C (crit = +20.8°C)
    temp2: +16.8°C (crit = +20.8°C)

    amdgpu-pci-0400
    Adapter: PCI adapter vddgfx: +0.72 V
    fan1: 768 RPM (min = 0 RPM, max = 3700 RPM)
    temp1: +39.0°C (crit = +94.0°C, hyst = -273.1°C)
    power1: 30.10 W (cap = 125.00 W)

    —
    _
    °v°
    /(_)\
    ^ ^ Mark LaPierre Registered Linux user No #267004
    https://linuxcounter.net/
    ****

  • …

    On a system that does not use any of the shim packages, would you recommend updating the grub2 and mokutil packages or holding off?

    Jon

  • Hi Johnny, thank you very much for clarification.

    You said that in the CentOS infrastructure only one server got the problem. What are the conditions that permit the breakage? There is a particular configuration (hw/sw) case that match always the problem or it is random?

    Thank you

    Il Sab 1 Ago 2020, 05:20 Johnny Hughes ha scritto:

  • First off, Johnny and all of the rest of the CentOS team, thank you for your efforts!

    Second, to all with this problem, I too experienced the issue (I posted on the CentOS-Devel list my findings).

  • I experienced the issue on a Dell Precision M6700 (UEFI, Core i7-3740QM)
    on the first attempt at update; I recovered by rolling back and reinstalling grub2-*; I then updated on the command line with dnf update and now that same hardware that crashed out the first time is working fine with the same package versions.

  • I posted, on 7/30, to CentOS-devel the following after my similar experience (booting an mSATA SSD on my M6700, with two 2.5 SATA drives
    (a 1TB and a 2TB), and the mSATA is not /dev/sda on this hardware, FWIW):

    “Moral of the story?

  • Well, I sure fat-fingered that command…. let’s try it again:

    [lowen@localhost ~]$ rpm -qa | grep ^kernel|grep 193.14
    kernel-headers-4.18.0-193.14.2.el8_2.x86_64
    kernel-devel-4.18.0-193.14.2.el8_2.x86_64
    kernel-modules-4.18.0-193.14.2.el8_2.x86_64
    kernel-tools-4.18.0-193.14.2.el8_2.x86_64
    kernel-core-4.18.0-193.14.2.el8_2.x86_64
    kernel-tools-libs-4.18.0-193.14.2.el8_2.x86_64
    kernel-4.18.0-193.14.2.el8_2.x86_64
    [lowen@localhost ~]$ uname -a Linux localhost.localdomain 4.18.0-193.14.2.el8_2.x86_64 #1 SMP Sun Jul
    26 03:54:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
    [lowen@localhost ~]$

  • I got 5 CentOS 8.1 VMs minimal install. When I did the update I got 1 that had dnf issues and corrupted rpm DB and another 1 that didn’t boot. Restore from snapshot and again update -> no issues. For me it looks more like a random issue.

    Best Regards, Strahil Nikolov

    На 1 август 2020 г. 18:28:09 GMT+03:00, Lamar Owen написа:

  • At 02:54 AM 8/1/2020, Alessandro Baggi wrote:

    I have two servers running CentOS 7 on apple hardware (one mac-mini and one mac server). They both failed to reboot a few days ago. So perhaps whatever anti-boot bug hit CentOS 8, also hit CentOS 7. I
    can’t tell what version got updated since the system simply fails to boot. I don’t even get a grub screen. I’ll have to rebuild the systems from scratch.

    David

  • You should be able to boot off of installation media into rescue mode, and downgrade the grub2* and/or shim* RPMs.

    -Greg

  • At 01:03 PM 8/1/2020, you wrote:

    This is a good idea, if I knew how to
    “downgrade…”. But in any event, I had decided to rebuild from scratch, which of course failed as soon as I did an yum update. So, I’m installing 7.8.2003 with no updates until I see the “all clear — updates will no longer make your system unbootable” message from the CentOS team.

    In my many years of blindly updating my installations, starting from the free Redhat distributions, through Whitehat and onto CentOS, this is the first disaster, and luckily, it didn’t hit all my systems. Let’s hope there aren’t many more.

    David

  • Don’t forget that EL 7 (non-UEFI systems) & 8 support booting from LVM snapshot (a.k.a BOOM Boot Manager) , so revert is 2 steps away:
    – boot from the snapshot (grub menu)
    – revert from the lvm snapshot
    – reboot and wait for the revert to complete

    Best Regards, Strahil Nikolov

    На 1 август 2020 г. 23:58:27 GMT+03:00, david написа:

  • Il 01/08/20 22:03, Greg Bailey ha scritto:
    selected packages (grub2,shim…) are already to the lowest version and the downgrade is not possibile, ending with “nothing to do”.

  • Ok .. We are running through some final testing now for CentOS Linux 8
    and CentOS Stream .. updates later today for EL8.

    For CentOS Linux 7 .. I just pushed the latest shim packages (we had to get these signed by Microsoft .. as do all distros that do shim. Microsoft is the official CA for secureboot.

    So in the next few hours, after the mirrors sync up .. you should be able to fix any EL7 machines.

    I’ll post here again once we have pushed the EL8 and CentOS Stream updates.

  • OK .. I have also now pushed the CentOS Linux 8 update .. you should see an update to SHIM .. the new versions are:

    PowerTools/x86_64/os/Packages/shim-unsigned-x64-15-8.el8.x86_64.rpm BaseOS/x86_64/os/Packages/shim-ia32-15-15.el8_2.x86_64.rpm BaseOS/x86_64/os/Packages/shim-x64-15-15.el8_2.x86_64.rpm

    For CentOS Linux 7 .. the new files are:

    x86_64/Packages/mokutil-15-8.el7.x86_64.rpm x86_64/Packages/shim-ia32-15-8.el7.x86_64.rpm x86_64/Packages/shim-unsigned-ia32-15-8.el7.x86_64.rpm x86_64/Packages/shim-unsigned-x64-15-8.el7.x86_64.rpm x86_64/Packages/shim-x64-15-8.el7.x86_64.rpm

    You need only replace the files you currently have installed, not install every file.

    Please report both positive and negative results.

    Thanks, Johnny Hughes

  • A previously afftected Aures nino POS-terminal boots just fine with CentOS 7 and shim 15.8.

    Now I will test an affected machine (Aures Twist POS terminal) that runs CentOS 8 (well, usually runs CentOS 8, but not now, since it does not boot … :=>

  • I updated my CentOS 7, and it’s awaiting a reboot. The version of shim says 15-7, which I presume is the bad one.

    What do I need to do? Obviously, a reboot will encounter the failure. Or, should I perform “yum update” periodically and wait for shim 15-8 to appear, and only after that do the reboot? Or is there some other remedy?

    And by the way, I’m amazed at the speed with which this problem was addressed and a fix provided. You guys work wonders.

    David

  • The affected machine running CentOS 8 boots fine after upgrading to shim 15.15

    Thanks for acting quickly, especially on a weekend.

  • It would be best IF the kernel and shim get installed at the same time if possible .. so I would install the kernel again (since it is not running yet) with the new shim.

    BTW .. it does not fail on every install .. so it might work w/o the new update. However, no need to chance it at this point. Wait until the new shim is there and install it and reinstall the kernel is my advise.

  • Thanks for the reports .. it is not the first weekend spent on this issue .. not even the first in a row :)

    The CentOS Stream update should be release in a few minutes unless we hit a snag in testing.

    Thanks, Johnny Hughes

  • At 06:10 AM 8/2/2020, you wrote:

    Here’s what I get…

    rpm -qa | grep shim shim-x64-15-7.el7_9.x86_64

    yum update Loaded plugins: fastestmirror, langpacks, priorities Loading mirror speeds from cached hostfile
    * base: mirror.fileplanet.com
    * epel: mirror.prgmr.com
    * extras: mirror.shastacoe.net
    * remi-php73: mirror.sjc02.svwh.net
    * remi-safe: mirror.sjc02.svwh.net
    * updates: CentOS-distro.1gservers.com
    373 packages excluded due to repository priority protections No packages marked for update

    Should I just wait for 15-8 to appear?

    David

  • Yes .. it should be on mirror.CentOS.org now .. you could change the repo where your updates come from. OR .. wait for that mirror to get updated.

  • At 06:37 AM 8/2/2020, Johnny Hughes wrote:

    I just did yum clean all yum update

    and 15-8 showed up. Maybe the ‘clean all’ did it, or maybe just showed up.

    I applied the update (yum update), rebooted and… no boot. Just a blank screen.

    Hardware is a Mac-Mini :-(

    This is not an essential machine, so I could re-install (from netinstall) if you think it’s worth the effort.

    Or what?

    David

  • Sorry for being so ignorant, but I don’t understand “just reinstall the kernel”. I don’t know how to translate that into a specific yum or rpm command.

    However, since this is a crash-and-burn system, I’m going back to a virgin install with netinstall of 7 2003. I’ll let you know what the results are.

    David

  • Sorry for being so ignorant, but I don’t understand “just reinstall the kernel”. I don’t know how to translate that into a specific yum or rpm command.

    However, since this is a crash-and-burn system, I’m going back to a virgin install with netinstall of 7 2003. I’ll let you know what the results are.

    ————-

  • I agree it is a lot of shorthand because of expectations. In the end we (the list) don’t know what you have on your system or what state it is in. In order to get that information to help we would need you to try the following:

    1. boot using a working USB/cdrom/netboot path and installer
    2. choose the rescue mode
    3. have the rescue mount the disks as local and chroot into the system. << if possible have the system also bring up networking >>

    Then yum list kernel shim grub2 mokutil

    It would also help to know which kind of Mac Mini it is (year, model, firmware versions). Apple changes the internal hardware of these things and how they boot so if there it may be that a particular model is more affected than others.

    —
    Stephen J Smoogen.

  • At 07:57 AM 8/2/2020, you wrote:

    How does one obtain that information? There’s nothing written on the box.

    David

  • I don’t have any access to a Mac Mini to help further than this general help:
    First I would search google for “how to obtain model version info on mac mini”

    https://support.apple.com/en-us/HT201894

    Then depending on how the mac mini is being booted (using the windows compatibility mode or some other method) I would look for howtos on how to update the firmware.

    After that it is time to learn the Mac magic key boot sequences. The Mac isn’t exactly a PC and its UEFI does not have a standard menu front end like many PC UEFI.

    https://support.apple.com/en-us/HT201255
    https://support.apple.com/en-us/HT202796

  • …

    Using dnf instead I would get my latest installed kernel version number:

    $ dnf list installed kernel
    Installed Packages
    kernel.x86_64 3.10.0-1127.el7 @base
    kernel.x86_64 3.10.0-1127.8.2.el7 @updates
    kernel.x86_64 3.10.0-1127.10.1.el7 @updates
    kernel.x86_64 3.10.0-1127.13.1.el7 @updates
    kernel.x86_64 3.10.0-1127.18.2.el7 @updates

    Then do:

    $ sudo dnf reinstall kernel*3.10.0-1127.18.2*

    HTH
    jon

  • The buggy package is the shim package. If you don’t have it on your system then you should not be affected.

    use the df command. If you are using EFI then it will report /boot/EFI
    as a partition. If it doesn’t then I would assume you are using BIOS.

    Since the problem is the shim package and you don’t seem to have it.. I would say exclusion is not needed.

  • Is it possible to bump the kernel version number to make sure the kernel gets re-installed on automated installs? Or would this break the compatibility with RHEL?

    P.

  • Well .. It would break ENVR with RHEL, right?

    RHEL is not bumping up their kernel version and they had the same issue.