Yum Update / Kernal Update Failed – Remove Or Repair

Home » CentOS » Yum Update / Kernal Update Failed – Remove Or Repair
CentOS 7 Comments

I have a VPS running C7. I ran a ‘yum update’ which included a kernel update. The yum update wasn’t a total success because mariadb updates failed. However everything else appeared to work.

However, when I rebooted the server it did not restart. I have managed to get the ISP’s support desk to boot the server by selecting the previous kernel, and I now have access to my box again.

As I will probably only have one chance to fix this (without having to get the ISP’s help again) I was wondering if there are any clear instructions on how to remove a failed kernel RPM update, returning the server to the state it was in prior to the YUM update?

7 thoughts on - Yum Update / Kernal Update Failed – Remove Or Repair

  • I have managed to find out what happened in the yum update and it turns out it was a mess. It looks like the server ran out of memory in the middle and things then started to fail. Any advice on how to recover from this would be greatly appreciated

    The log below shows:

    [root@vps2 ~]# yum history info 22
    Loaded plugins: fastestmirror, langpacks Repository mariadb is listed more than once in the configuration Transaction ID : 22
    Begin time : Sun Jan 19 18:15:33 2020
    Begin rpmdb : 795:6c5cd82e3ada047c7071fea1524f41fd13fab6a1
    End time : 19:19:32 2020 (63 minutes)
    End rpmdb : 801:6d1b278ccbeef6f68ff494cb28eaed5c7be4124b User : root
    Return-Code : Failure: 1
    Command Line : update Transaction performed with:
    Installed rpm-4.11.3-40.el7.x86_64 @anaconda
    Installed yum-3.4.3-163.el7.CentOS.noarch @anaconda
    Installed yum-plugin-fastestmirror-1.1.31-52.el7.noarch @anaconda Packages Altered:
    Updated MariaDB-client-10.4.10-1.el7.CentOS.x86_64 @mariadb
    Update 10.4.11-1.el7.CentOS.x86_64 @mariadb
    Updated MariaDB-common-10.4.10-1.el7.CentOS.x86_64 @mariadb
    Update 10.4.11-1.el7.CentOS.x86_64 @mariadb
    Updated MariaDB-compat-10.4.10-1.el7.CentOS.x86_64 @mariadb
    Update 10.4.11-1.el7.CentOS.x86_64 @mariadb
    ** Updated MariaDB-server-10.4.10-1.el7.CentOS.x86_64 @mariadb Loading mirror speeds from cached hostfile
    * base: mirror.as29550.net
    * epel: http://www.mirrorservice.org
    * extras: http://www.mirrorservice.org
    * remi-php73: mirrors.ukfast.co.uk
    * remi-safe: mirrors.ukfast.co.uk
    * updates: http://www.mirrorservice.org
    ** Update 10.4.11-1.el7.CentOS.x86_64 @?mariadb
    ** Updated MariaDB-shared-10.4.10-1.el7.CentOS.x86_64 @mariadb
    Update 10.4.11-1.el7.CentOS.x86_64 @mariadb
    ** Updated ca-certificates-2018.2.22-70.0.el7_5.noarch @anaconda
    ** Update 2019.2.32-76.el7_7.noarch @?updates
    Updated certbot-0.39.0-1.el7.noarch @epel
    Update 1.0.0-1.el7.noarch @epel
    Updated clamav-0.101.4-1.el7.x86_64 @epel
    Update 0.101.5-1.el7.x86_64 @epel
    Updated clamav-data-0.101.4-1.el7.noarch @epel
    Update 0.101.5-1.el7.noarch @epel
    Updated clamav-devel-0.101.4-1.el7.x86_64 @epel
    Update 0.101.5-1.el7.x86_64 @epel
    Updated clamav-filesystem-0.101.4-1.el7.noarch @epel
    Update 0.101.5-1.el7.noarch @epel
    Updated clamav-lib-0.101.4-1.el7.x86_64 @epel
    Update 0.101.5-1.el7.x86_64 @epel
    Updated clamav-update-0.101.4-1.el7.x86_64 @epel
    Update 0.101.5-1.el7.x86_64 @epel
    Updated clamd-0.101.4-1.el7.x86_64 @epel
    Update 0.101.5-1.el7.x86_64 @epel
    Updated curl-7.29.0-54.el7.x86_64 @anaconda
    Update 7.29.0-54.el7_7.1.x86_64 @updates
    Updated fail2ban-0.9.7-1.el7.noarch @epel
    Update 0.10.4-1.el7.noarch @epel
    Updated fail2ban-firewalld-0.9.7-1.el7.noarch @epel
    Update 0.10.4-1.el7.noarch @epel
    Updated fail2ban-sendmail-0.9.7-1.el7.noarch @epel
    Update 0.10.4-1.el7.noarch @epel
    Updated fail2ban-server-0.9.7-1.el7.noarch @epel
    Update 0.10.4-1.el7.noarch @epel
    Updated fail2ban-systemd-0.9.7-1.el7.noarch @epel
    Update 0.10.4-1.el7.noarch @epel
    Updated fribidi-1.0.2-1.el7.x86_64 @anaconda
    Update 1.0.2-1.el7_7.1.x86_64 @updates
    Updated iproute-4.11.0-25.el7.x86_64 @anaconda
    Update 4.11.0-25.el7_7.2.x86_64 @updates
    Install kernel-3.10.0-1062.9.1.el7.x86_64 @updates
    Updated kernel-headers-3.10.0-1062.4.3.el7.x86_64 @updates
    Update 3.10.0-1062.9.1.el7.x86_64 @updates
    Updated kernel-tools-3.10.0-1062.4.3.el7.x86_64 @updates
    Update 3.10.0-1062.9.1.el7.x86_64 @updates
    Updated kernel-tools-libs-3.10.0-1062.4.3.el7.x86_64 @updates
    Update 3.10.0-1062.9.1.el7.x86_64 @updates
    Updated libblkid-2.23.2-61.el7.x86_64 @anaconda
    Update 2.23.2-61.el7_7.1.x86_64 @updates
    Updated libcurl-7.29.0-54.el7.x86_64 @anaconda
    Update 7.29.0-54.el7_7.1.x86_64 @updates
    Updated libgs-9.25-2.el7_7.2.x86_64 @updates
    Update 9.25-2.el7_7.3.x86_64 @updates
    Updated libmount-2.23.2-61.el7.x86_64 @anaconda
    Update 2.23.2-61.el7_7.1.x86_64 @updates
    Updated libsmartcols-2.23.2-61.el7.x86_64 @anaconda
    Update 2.23.2-61.el7_7.1.x86_64 @updates
    Updated libsss_idmap-1.16.4-21.el7.x86_64 @anaconda
    Update 1.16.4-21.el7_7.1.x86_64 @updates
    Updated libsss_nss_idmap-1.16.4-21.el7.x86_64 @anaconda
    Update 1.16.4-21.el7_7.1.x86_64 @updates
    Updated libuuid-2.23.2-61.el7.x86_64 @anaconda
    Update 2.23.2-61.el7_7.1.x86_64 @updates
    Updated libwebp7-1.0.2-1.el7.remi.x86_64 @remi-safe
    Update 1.0.3-1.el7.remi.x86_64 @remi-safe
    ** Updated microcode_ctl-2:2.1-53.3.el7_7.x86_64 @updates
    Update 2:2.1-53.7.el7_7.x86_64 @updates
    Updated nss-3.44.0-4.el7.x86_64 @anaconda
    Update 3.44.0-7.el7_7.x86_64 @updates
    Updated nss-softokn-3.44.0-5.el7.x86_64 @anaconda
    Update 3.44.0-8.el7_7.x86_64 @updates
    Updated nss-softokn-freebl-3.44.0-5.el7.x86_64 @anaconda
    Update 3.44.0-8.el7_7.x86_64 @updates
    Updated nss-sysinit-3.44.0-4.el7.x86_64 @anaconda
    Update 3.44.0-7.el7_7.x86_64 @updates
    Updated nss-tools-3.44.0-4.el7.x86_64 @anaconda
    Update 3.44.0-7.el7_7.x86_64 @updates
    Updated nss-util-3.44.0-3.el7.x86_64 @anaconda
    Update 3.44.0-4.el7_7.x86_64 @updates
    Updated numactl-libs-2.0.12-3.el7.x86_64 @anaconda
    Update 2.0.12-3.el7_7.1.x86_64 @updates
    Updated pgdg-redhat-repo-42.0-5.noarch @/pgdg-CentOS10-10-2.noarch
    Update 42.0-6.noarch @pgdg10
    Updated php-7.3.12-1.el7.remi.x86_64 @remi-php73
    Update 7.3.13-1.el7.remi.x86_64 @remi-php73
    Updated php-cli-7.3.12-1.el7.remi.x86_64 @remi-php73
    Update 7.3.13-1.el7.remi.x86_64 @remi-php73
    Updated php-common-7.3.12-1.el7.remi.x86_64 @remi-php73
    Update 7.3.13-1.el7.remi.x86_64 @remi-php73
    Updated php-gd-7.3.12-1.el7.remi.x86_64 @remi-php73
    Update 7.3.13-1.el7.remi.x86_64 @remi-php73
    Updated php-json-7.3.12-1.el7.remi.x86_64 @remi-php73
    Update 7.3.13-1.el7.remi.x86_64 @remi-php73
    Updated php-ldap-7.3.12-1.el7.remi.x86_64 @remi-php73
    Update 7.3.13-1.el7.remi.x86_64 @remi-php73
    Updated php-mysqlnd-7.3.12-1.el7.remi.x86_64 @remi-php73
    Update 7.3.13-1.el7.remi.x86_64 @remi-php73
    Updated php-odbc-7.3.12-1.el7.remi.x86_64 @remi-php73
    Update 7.3.13-1.el7.remi.x86_64 @remi-php73
    Updated php-pdo-7.3.12-1.el7.remi.x86_64 @remi-php73
    Update 7.3.13-1.el7.remi.x86_64 @remi-php73
    ** Updated php-pear-1:1.10.10-2.el7.remi.noarch @remi-php70
    ** Update 1:1.10.10-4.el7.remi.noarch @?remi-php73
    Updated php-pgsql-7.3.12-1.el7.remi.x86_64 @remi-php73
    Update 7.3.13-1.el7.remi.x86_64 @remi-php73
    Updated php-process-7.3.12-1.el7.remi.x86_64 @remi-php73
    Update 7.3.13-1.el7.remi.x86_64 @remi-php73
    Updated php-xml-7.3.12-1.el7.remi.x86_64 @remi-php73
    Update 7.3.13-1.el7.remi.x86_64 @remi-php73
    Updated php-xmlrpc-7.3.12-1.el7.remi.x86_64 @remi-php73
    Update 7.3.13-1.el7.remi.x86_64 @remi-php73
    Updated PostgreSQL10-10.11-1PGDG.rhel7.x86_64 @pgdg10
    Update 10.11-2PGDG.rhel7.x86_64 @pgdg10
    Updated PostgreSQL10-contrib-10.11-1PGDG.rhel7.x86_64 @pgdg10
    Update 10.11-2PGDG.rhel7.x86_64 @pgdg10
    Updated PostgreSQL10-libs-10.11-1PGDG.rhel7.x86_64 @pgdg10
    Update 10.11-2PGDG.rhel7.x86_64 @pgdg10
    ** Updated PostgreSQL10-server-10.11-1PGDG.rhel7.x86_64 @pgdg10
    Update 10.11-2PGDG.rhel7.x86_64 @pgdg10
    ** Updated proftpd-1.3.5e-7.el7.x86_64 @epel
    Update 1.3.5e-8.el7.x86_64 @epel
    Updated python-perf-3.10.0-1062.4.3.el7.x86_64 @updates
    Update 3.10.0-1062.9.1.el7.x86_64 @updates
    Updated python-requests-2.6.0-7.el7_7.noarch @updates
    Update 2.6.0-8.el7_7.noarch @updates
    Updated python2-acme-0.39.0-1.el7.noarch @epel
    Update 1.0.0-1.el7.noarch @epel
    Updated python2-certbot-0.39.0-1.el7.noarch @epel
    Update 1.0.0-1.el7.noarch @epel
    Updated python2-certbot-apache-0.39.0-1.el7.noarch @epel
    Update 1.0.0-1.el7.noarch @epel
    Updated selinux-policy-3.13.1-252.el7.1.noarch @updates
    Update 3.13.1-252.el7_7.6.noarch @updates
    Updated selinux-policy-targeted-3.13.1-252.el7.1.noarch @updates
    Update 3.13.1-252.el7_7.6.noarch @updates
    Updated sos-3.7-7.el7.CentOS.noarch @updates
    Update 3.7-10.el7.CentOS.noarch @updates
    ** Updated sssd-client-1.16.4-21.el7.x86_64 @anaconda
    Update 1.16.4-21.el7_7.1.x86_64 @updates
    Updated tcpdump-14:4.9.2-4.el7.x86_64 @anaconda
    Update 14:4.9.2-4.el7_7.1.x86_64 @updates
    Updated util-linux-2.23.2-61.el7.x86_64 @anaconda
    Update 2.23.2-61.el7_7.1.x86_64 @updates Scriptlet output:
    1 error: Couldn’t fork %post(libsss_idmap-1.16.4-21.el7_7.1.x86_64): Cannot allocate memory
    2 error: Couldn’t fork %post(libsss_nss_idmap-1.16.4-21.el7_7.1.x86_64): Cannot allocate memory
    3 error: Couldn’t fork %pre(ca-certificates-2019.2.32-76.el7_7.noarch): Cannot allocate memory
    4 error: ca-certificates-2019.2.32-76.el7_7.noarch: install failed
    5 error: Couldn’t fork %post(certbot-1.0.0-1.el7.noarch): Cannot allocate memory
    6 error: Couldn’t fork %pre(MariaDB-server-10.4.11-1.el7.CentOS.x86_64): Cannot allocate memory
    7 error: MariaDB-server-10.4.11-1.el7.CentOS.x86_64: install failed
    8 error: Couldn’t fork %post(sssd-client-1.16.4-21.el7_7.1.x86_64): Cannot allocate memory
    9 error: Couldn’t fork %post(kernel-tools-3.10.0-1062.9.1.el7.x86_64): Cannot allocate memory
    10 error: Couldn’t fork %pre(php-pear-1:1.10.10-4.el7.remi.noarch): Cannot allocate memory
    11 error: php-pear-1:1.10.10-4.el7.remi.noarch: install failed
    12 warning: /etc/clamd.d/scan.conf created as /etc/clamd.d/scan.conf.rpmnew
    13 error: Couldn’t fork %triggerin(microcode_ctl-2:2.1-53.7.el7_7.x86_64): Cannot allocate memory
    14 error: php-pear-1:1.10.10-2.el7.remi.noarch: erase skipped
    15 error: MariaDB-server-10.4.10-1.el7.CentOS.x86_64: erase skipped
    16 error: Couldn’t fork %preun(MariaDB-shared-10.4.10-1.el7.CentOS.x86_64): Cannot allocate memory
    17 error: MariaDB-shared-10.4.10-1.el7.CentOS.x86_64: erase failed
    18 error: Couldn’t fork %postun(MariaDB-compat-10.4.10-1.el7.CentOS.x86_64): Cannot allocate memory
    19 error: ca-certificates-2018.2.22-70.0.el7_5.noarch: erase skipped
    20 error: Couldn’t fork %preun(microcode_ctl-2:2.1-53.3.el7_7.x86_64): Cannot allocate memory
    21 error: microcode_ctl-2:2.1-53.3.el7_7.x86_64: erase failed
    22 error: Couldn’t fork %preun(PostgreSQL10-server-10.11-1PGDG.rhel7.x86_64): Cannot allocate memory
    23 error: PostgreSQL10-server-10.11-1PGDG.rhel7.x86_64: erase failed
    24 error: Couldn’t fork %postun(libmount-2.23.2-61.el7.x86_64): Cannot allocate memory
    25 error: Couldn’t fork %preun(sssd-client-1.16.4-21.el7.x86_64): Cannot allocate memory
    26 error: sssd-client-1.16.4-21.el7.x86_64: erase failed
    27 error: Couldn’t fork %postun(libblkid-2.23.2-61.el7.x86_64): Cannot allocate memory
    28 error: Couldn’t fork %postun(PostgreSQL10-10.11-1PGDG.rhel7.x86_64): Cannot allocate memory
    29 error: Couldn’t fork %postun(clamav-lib-0.101.4-1.el7.x86_64): Cannot allocate memory
    30 error: Couldn’t fork %postun(kernel-tools-3.10.0-1062.4.3.el7.x86_64): Cannot allocate memory
    31 error: Couldn’t fork %postun(libcurl-7.29.0-54.el7.x86_64): Cannot allocate memory
    32 error: Couldn’t fork %postun(nss-3.44.0-4.el7.x86_64): Cannot allocate memory
    33 error: Couldn’t fork %postun(nss-softokn-3.44.0-5.el7.x86_64): Cannot allocate memory
    34 error: Couldn’t fork %postun(nss-util-3.44.0-3.el7.x86_64): Cannot allocate memory
    35 error: Couldn’t fork %postun(PostgreSQL10-libs-10.11-1PGDG.rhel7.x86_64): Cannot allocate memory
    36 error: Couldn’t fork %postun(libuuid-2.23.2-61.el7.x86_64): Cannot allocate memory
    37 error: Couldn’t fork %postun(libsss_idmap-1.16.4-21.el7.x86_64): Cannot allocate memory
    38 error: Couldn’t fork %postun(libsss_nss_idmap-1.16.4-21.el7.x86_64): Cannot allocate memory
    39 error: Couldn’t fork %postun(libsmartcols-2.23.2-61.el7.x86_64): Cannot allocate memory
    40 error: Couldn’t fork %postun(libwebp7-1.0.2-1.el7.remi.x86_64): Cannot allocate memory
    41 error: Couldn’t fork %postun(fribidi-1.0.2-1.el7.x86_64): Cannot allocate memory
    42 error: Couldn’t fork %preun(proftpd-1.3.5e-7.el7.x86_64): Cannot allocate memory
    43 error: proftpd-1.3.5e-7.el7.x86_64: erase failed
    44 error: Couldn’t fork %postun(numactl-libs-2.0.12-3.el7.x86_64): Cannot allocate memory
    45 error: Couldn’t fork %posttrans(kernel-3.10.0-1062.9.1.el7.x86_64): Cannot allocate memory
    46 error: Couldn’t fork %posttrans(microcode_ctl-2:2.1-53.7.el7_7.x86_64): Cannot allocate memory history info
    [root@vps2 ~]#

  • I may sound old school but my suggestion would be to forget yum aka dnf for now and just fix the real stuff using rpm.

    I always considered yum a wrapper for rpm (I wish it was) and was able to fix a system with rpm alone and later make yum behave again.

    It usually boils down to analyzing what is in the rpm db and what is on the filesystem actually. You can then remove invalid rpm db content as well as wrong filesystem content. It’s a bit of work and you may create helper scripts as needed but you can fix almost everything that way.

    Regards, Simon

  • Before you try the update again, you’ll have to fix the reason for the failure – add memory, or at least add a swap file.

    You could check with rpm -qa –last | head -20
    which the latest packages are that were installed.

    If the rpm database is corrupted, rebuild it with rpm –rebuilddb

    You can reinstall packages that may not be completely installed, using rpm –reinstall PACKAGE_FILE

    I’d rpm –erase the kernel package that does not boot.

    HTH, Kay

  • Well, there may be more to check and above steps may not help.

    Do you have duplicates in rpmdb? Which one of the duplicates are already on the filesystem?

    Regards, Simon

  • I do not believe that I have any duplicate RPM’s installed, and I do not believe that the database is actually wrong. I believe that it is purely that the installs did not complete successfully.

    I have managed to remove the latest kernel RPM and ‘yum update kernel*’ re-applied the update without errors.

    I am going to go through the list of failed RPM’s and to a ‘rpm –reinstall’ for each one in turn and hopefully they will work too.

    Before doing that I will do as Kay suggested and find out why I had the memory problem in the first place.