Yum Update / Kernal Update Failed – Remove Or Repair
I have a VPS running C7. I ran a ‘yum update’ which included a kernel update. The yum update wasn’t a total success because mariadb updates failed. However everything else appeared to work.
However, when I rebooted the server it did not restart. I have managed to get the ISP’s support desk to boot the server by selecting the previous kernel, and I now have access to my box again.
As I will probably only have one chance to fix this (without having to get the ISP’s help again) I was wondering if there are any clear instructions on how to remove a failed kernel RPM update, returning the server to the state it was in prior to the YUM update?
7 thoughts on - Yum Update / Kernal Update Failed – Remove Or Repair
I have managed to find out what happened in the yum update and it turns out it was a mess. It looks like the server ran out of memory in the middle and things then started to fail. Any advice on how to recover from this would be greatly appreciated
The log below shows:
[root@vps2 ~]# yum history info 22
Loaded plugins: fastestmirror, langpacks Repository mariadb is listed more than once in the configuration Transaction ID : 22
Begin time : Sun Jan 19 18:15:33 2020
Begin rpmdb : 795:6c5cd82e3ada047c7071fea1524f41fd13fab6a1
End time : 19:19:32 2020 (63 minutes)
End rpmdb : 801:6d1b278ccbeef6f68ff494cb28eaed5c7be4124b User : root
Return-Code : Failure: 1
Command Line : update Transaction performed with:
Installed rpm-4.11.3-40.el7.x86_64 @anaconda
Installed yum-3.4.3-163.el7.CentOS.noarch @anaconda
Installed yum-plugin-fastestmirror-1.1.31-52.el7.noarch @anaconda Packages Altered:
Updated MariaDB-client-10.4.10-1.el7.CentOS.x86_64 @mariadb
Update 10.4.11-1.el7.CentOS.x86_64 @mariadb
Updated MariaDB-common-10.4.10-1.el7.CentOS.x86_64 @mariadb
Update 10.4.11-1.el7.CentOS.x86_64 @mariadb
Updated MariaDB-compat-10.4.10-1.el7.CentOS.x86_64 @mariadb
Update 10.4.11-1.el7.CentOS.x86_64 @mariadb
** Updated MariaDB-server-10.4.10-1.el7.CentOS.x86_64 @mariadb Loading mirror speeds from cached hostfile
* base: mirror.as29550.net
* epel: http://www.mirrorservice.org
* extras: http://www.mirrorservice.org
* remi-php73: mirrors.ukfast.co.uk
* remi-safe: mirrors.ukfast.co.uk
* updates: http://www.mirrorservice.org
** Update 10.4.11-1.el7.CentOS.x86_64 @?mariadb
** Updated MariaDB-shared-10.4.10-1.el7.CentOS.x86_64 @mariadb
Update 10.4.11-1.el7.CentOS.x86_64 @mariadb
** Updated ca-certificates-2018.2.22-70.0.el7_5.noarch @anaconda
** Update 2019.2.32-76.el7_7.noarch @?updates
Updated certbot-0.39.0-1.el7.noarch @epel
Update 1.0.0-1.el7.noarch @epel
Updated clamav-0.101.4-1.el7.x86_64 @epel
Update 0.101.5-1.el7.x86_64 @epel
Updated clamav-data-0.101.4-1.el7.noarch @epel
Update 0.101.5-1.el7.noarch @epel
Updated clamav-devel-0.101.4-1.el7.x86_64 @epel
Update 0.101.5-1.el7.x86_64 @epel
Updated clamav-filesystem-0.101.4-1.el7.noarch @epel
Update 0.101.5-1.el7.noarch @epel
Updated clamav-lib-0.101.4-1.el7.x86_64 @epel
Update 0.101.5-1.el7.x86_64 @epel
Updated clamav-update-0.101.4-1.el7.x86_64 @epel
Update 0.101.5-1.el7.x86_64 @epel
Updated clamd-0.101.4-1.el7.x86_64 @epel
Update 0.101.5-1.el7.x86_64 @epel
Updated curl-7.29.0-54.el7.x86_64 @anaconda
Update 7.29.0-54.el7_7.1.x86_64 @updates
Updated fail2ban-0.9.7-1.el7.noarch @epel
Update 0.10.4-1.el7.noarch @epel
Updated fail2ban-firewalld-0.9.7-1.el7.noarch @epel
Update 0.10.4-1.el7.noarch @epel
Updated fail2ban-sendmail-0.9.7-1.el7.noarch @epel
Update 0.10.4-1.el7.noarch @epel
Updated fail2ban-server-0.9.7-1.el7.noarch @epel
Update 0.10.4-1.el7.noarch @epel
Updated fail2ban-systemd-0.9.7-1.el7.noarch @epel
Update 0.10.4-1.el7.noarch @epel
Updated fribidi-1.0.2-1.el7.x86_64 @anaconda
Update 1.0.2-1.el7_7.1.x86_64 @updates
Updated iproute-4.11.0-25.el7.x86_64 @anaconda
Update 4.11.0-25.el7_7.2.x86_64 @updates
Install kernel-3.10.0-1062.9.1.el7.x86_64 @updates
Updated kernel-headers-3.10.0-1062.4.3.el7.x86_64 @updates
Update 3.10.0-1062.9.1.el7.x86_64 @updates
Updated kernel-tools-3.10.0-1062.4.3.el7.x86_64 @updates
Update 3.10.0-1062.9.1.el7.x86_64 @updates
Updated kernel-tools-libs-3.10.0-1062.4.3.el7.x86_64 @updates
Update 3.10.0-1062.9.1.el7.x86_64 @updates
Updated libblkid-2.23.2-61.el7.x86_64 @anaconda
Update 2.23.2-61.el7_7.1.x86_64 @updates
Updated libcurl-7.29.0-54.el7.x86_64 @anaconda
Update 7.29.0-54.el7_7.1.x86_64 @updates
Updated libgs-9.25-2.el7_7.2.x86_64 @updates
Update 9.25-2.el7_7.3.x86_64 @updates
Updated libmount-2.23.2-61.el7.x86_64 @anaconda
Update 2.23.2-61.el7_7.1.x86_64 @updates
Updated libsmartcols-2.23.2-61.el7.x86_64 @anaconda
Update 2.23.2-61.el7_7.1.x86_64 @updates
Updated libsss_idmap-1.16.4-21.el7.x86_64 @anaconda
Update 1.16.4-21.el7_7.1.x86_64 @updates
Updated libsss_nss_idmap-1.16.4-21.el7.x86_64 @anaconda
Update 1.16.4-21.el7_7.1.x86_64 @updates
Updated libuuid-2.23.2-61.el7.x86_64 @anaconda
Update 2.23.2-61.el7_7.1.x86_64 @updates
Updated libwebp7-1.0.2-1.el7.remi.x86_64 @remi-safe
Update 1.0.3-1.el7.remi.x86_64 @remi-safe
** Updated microcode_ctl-2:2.1-53.3.el7_7.x86_64 @updates
Update 2:2.1-53.7.el7_7.x86_64 @updates
Updated nss-3.44.0-4.el7.x86_64 @anaconda
Update 3.44.0-7.el7_7.x86_64 @updates
Updated nss-softokn-3.44.0-5.el7.x86_64 @anaconda
Update 3.44.0-8.el7_7.x86_64 @updates
Updated nss-softokn-freebl-3.44.0-5.el7.x86_64 @anaconda
Update 3.44.0-8.el7_7.x86_64 @updates
Updated nss-sysinit-3.44.0-4.el7.x86_64 @anaconda
Update 3.44.0-7.el7_7.x86_64 @updates
Updated nss-tools-3.44.0-4.el7.x86_64 @anaconda
Update 3.44.0-7.el7_7.x86_64 @updates
Updated nss-util-3.44.0-3.el7.x86_64 @anaconda
Update 3.44.0-4.el7_7.x86_64 @updates
Updated numactl-libs-2.0.12-3.el7.x86_64 @anaconda
Update 2.0.12-3.el7_7.1.x86_64 @updates
Updated pgdg-redhat-repo-42.0-5.noarch @/pgdg-CentOS10-10-2.noarch
Update 42.0-6.noarch @pgdg10
Updated php-7.3.12-1.el7.remi.x86_64 @remi-php73
Update 7.3.13-1.el7.remi.x86_64 @remi-php73
Updated php-cli-7.3.12-1.el7.remi.x86_64 @remi-php73
Update 7.3.13-1.el7.remi.x86_64 @remi-php73
Updated php-common-7.3.12-1.el7.remi.x86_64 @remi-php73
Update 7.3.13-1.el7.remi.x86_64 @remi-php73
Updated php-gd-7.3.12-1.el7.remi.x86_64 @remi-php73
Update 7.3.13-1.el7.remi.x86_64 @remi-php73
Updated php-json-7.3.12-1.el7.remi.x86_64 @remi-php73
Update 7.3.13-1.el7.remi.x86_64 @remi-php73
Updated php-ldap-7.3.12-1.el7.remi.x86_64 @remi-php73
Update 7.3.13-1.el7.remi.x86_64 @remi-php73
Updated php-mysqlnd-7.3.12-1.el7.remi.x86_64 @remi-php73
Update 7.3.13-1.el7.remi.x86_64 @remi-php73
Updated php-odbc-7.3.12-1.el7.remi.x86_64 @remi-php73
Update 7.3.13-1.el7.remi.x86_64 @remi-php73
Updated php-pdo-7.3.12-1.el7.remi.x86_64 @remi-php73
Update 7.3.13-1.el7.remi.x86_64 @remi-php73
** Updated php-pear-1:1.10.10-2.el7.remi.noarch @remi-php70
** Update 1:1.10.10-4.el7.remi.noarch @?remi-php73
Updated php-pgsql-7.3.12-1.el7.remi.x86_64 @remi-php73
Update 7.3.13-1.el7.remi.x86_64 @remi-php73
Updated php-process-7.3.12-1.el7.remi.x86_64 @remi-php73
Update 7.3.13-1.el7.remi.x86_64 @remi-php73
Updated php-xml-7.3.12-1.el7.remi.x86_64 @remi-php73
Update 7.3.13-1.el7.remi.x86_64 @remi-php73
Updated php-xmlrpc-7.3.12-1.el7.remi.x86_64 @remi-php73
Update 7.3.13-1.el7.remi.x86_64 @remi-php73
Updated PostgreSQL10-10.11-1PGDG.rhel7.x86_64 @pgdg10
Update 10.11-2PGDG.rhel7.x86_64 @pgdg10
Updated PostgreSQL10-contrib-10.11-1PGDG.rhel7.x86_64 @pgdg10
Update 10.11-2PGDG.rhel7.x86_64 @pgdg10
Updated PostgreSQL10-libs-10.11-1PGDG.rhel7.x86_64 @pgdg10
Update 10.11-2PGDG.rhel7.x86_64 @pgdg10
** Updated PostgreSQL10-server-10.11-1PGDG.rhel7.x86_64 @pgdg10
Update 10.11-2PGDG.rhel7.x86_64 @pgdg10
** Updated proftpd-1.3.5e-7.el7.x86_64 @epel
Update 1.3.5e-8.el7.x86_64 @epel
Updated python-perf-3.10.0-1062.4.3.el7.x86_64 @updates
Update 3.10.0-1062.9.1.el7.x86_64 @updates
Updated python-requests-2.6.0-7.el7_7.noarch @updates
Update 2.6.0-8.el7_7.noarch @updates
Updated python2-acme-0.39.0-1.el7.noarch @epel
Update 1.0.0-1.el7.noarch @epel
Updated python2-certbot-0.39.0-1.el7.noarch @epel
Update 1.0.0-1.el7.noarch @epel
Updated python2-certbot-apache-0.39.0-1.el7.noarch @epel
Update 1.0.0-1.el7.noarch @epel
Updated selinux-policy-3.13.1-252.el7.1.noarch @updates
Update 3.13.1-252.el7_7.6.noarch @updates
Updated selinux-policy-targeted-3.13.1-252.el7.1.noarch @updates
Update 3.13.1-252.el7_7.6.noarch @updates
Updated sos-3.7-7.el7.CentOS.noarch @updates
Update 3.7-10.el7.CentOS.noarch @updates
** Updated sssd-client-1.16.4-21.el7.x86_64 @anaconda
Update 1.16.4-21.el7_7.1.x86_64 @updates
Updated tcpdump-14:4.9.2-4.el7.x86_64 @anaconda
Update 14:4.9.2-4.el7_7.1.x86_64 @updates
Updated util-linux-2.23.2-61.el7.x86_64 @anaconda
Update 2.23.2-61.el7_7.1.x86_64 @updates Scriptlet output:
1 error: Couldn’t fork %post(libsss_idmap-1.16.4-21.el7_7.1.x86_64): Cannot allocate memory
2 error: Couldn’t fork %post(libsss_nss_idmap-1.16.4-21.el7_7.1.x86_64): Cannot allocate memory
3 error: Couldn’t fork %pre(ca-certificates-2019.2.32-76.el7_7.noarch): Cannot allocate memory
4 error: ca-certificates-2019.2.32-76.el7_7.noarch: install failed
5 error: Couldn’t fork %post(certbot-1.0.0-1.el7.noarch): Cannot allocate memory
6 error: Couldn’t fork %pre(MariaDB-server-10.4.11-1.el7.CentOS.x86_64): Cannot allocate memory
7 error: MariaDB-server-10.4.11-1.el7.CentOS.x86_64: install failed
8 error: Couldn’t fork %post(sssd-client-1.16.4-21.el7_7.1.x86_64): Cannot allocate memory
9 error: Couldn’t fork %post(kernel-tools-3.10.0-1062.9.1.el7.x86_64): Cannot allocate memory
10 error: Couldn’t fork %pre(php-pear-1:1.10.10-4.el7.remi.noarch): Cannot allocate memory
11 error: php-pear-1:1.10.10-4.el7.remi.noarch: install failed
12 warning: /etc/clamd.d/scan.conf created as /etc/clamd.d/scan.conf.rpmnew
13 error: Couldn’t fork %triggerin(microcode_ctl-2:2.1-53.7.el7_7.x86_64): Cannot allocate memory
14 error: php-pear-1:1.10.10-2.el7.remi.noarch: erase skipped
15 error: MariaDB-server-10.4.10-1.el7.CentOS.x86_64: erase skipped
16 error: Couldn’t fork %preun(MariaDB-shared-10.4.10-1.el7.CentOS.x86_64): Cannot allocate memory
17 error: MariaDB-shared-10.4.10-1.el7.CentOS.x86_64: erase failed
18 error: Couldn’t fork %postun(MariaDB-compat-10.4.10-1.el7.CentOS.x86_64): Cannot allocate memory
19 error: ca-certificates-2018.2.22-70.0.el7_5.noarch: erase skipped
20 error: Couldn’t fork %preun(microcode_ctl-2:2.1-53.3.el7_7.x86_64): Cannot allocate memory
21 error: microcode_ctl-2:2.1-53.3.el7_7.x86_64: erase failed
22 error: Couldn’t fork %preun(PostgreSQL10-server-10.11-1PGDG.rhel7.x86_64): Cannot allocate memory
23 error: PostgreSQL10-server-10.11-1PGDG.rhel7.x86_64: erase failed
24 error: Couldn’t fork %postun(libmount-2.23.2-61.el7.x86_64): Cannot allocate memory
25 error: Couldn’t fork %preun(sssd-client-1.16.4-21.el7.x86_64): Cannot allocate memory
26 error: sssd-client-1.16.4-21.el7.x86_64: erase failed
27 error: Couldn’t fork %postun(libblkid-2.23.2-61.el7.x86_64): Cannot allocate memory
28 error: Couldn’t fork %postun(PostgreSQL10-10.11-1PGDG.rhel7.x86_64): Cannot allocate memory
29 error: Couldn’t fork %postun(clamav-lib-0.101.4-1.el7.x86_64): Cannot allocate memory
30 error: Couldn’t fork %postun(kernel-tools-3.10.0-1062.4.3.el7.x86_64): Cannot allocate memory
31 error: Couldn’t fork %postun(libcurl-7.29.0-54.el7.x86_64): Cannot allocate memory
32 error: Couldn’t fork %postun(nss-3.44.0-4.el7.x86_64): Cannot allocate memory
33 error: Couldn’t fork %postun(nss-softokn-3.44.0-5.el7.x86_64): Cannot allocate memory
34 error: Couldn’t fork %postun(nss-util-3.44.0-3.el7.x86_64): Cannot allocate memory
35 error: Couldn’t fork %postun(PostgreSQL10-libs-10.11-1PGDG.rhel7.x86_64): Cannot allocate memory
36 error: Couldn’t fork %postun(libuuid-2.23.2-61.el7.x86_64): Cannot allocate memory
37 error: Couldn’t fork %postun(libsss_idmap-1.16.4-21.el7.x86_64): Cannot allocate memory
38 error: Couldn’t fork %postun(libsss_nss_idmap-1.16.4-21.el7.x86_64): Cannot allocate memory
39 error: Couldn’t fork %postun(libsmartcols-2.23.2-61.el7.x86_64): Cannot allocate memory
40 error: Couldn’t fork %postun(libwebp7-1.0.2-1.el7.remi.x86_64): Cannot allocate memory
41 error: Couldn’t fork %postun(fribidi-1.0.2-1.el7.x86_64): Cannot allocate memory
42 error: Couldn’t fork %preun(proftpd-1.3.5e-7.el7.x86_64): Cannot allocate memory
43 error: proftpd-1.3.5e-7.el7.x86_64: erase failed
44 error: Couldn’t fork %postun(numactl-libs-2.0.12-3.el7.x86_64): Cannot allocate memory
45 error: Couldn’t fork %posttrans(kernel-3.10.0-1062.9.1.el7.x86_64): Cannot allocate memory
46 error: Couldn’t fork %posttrans(microcode_ctl-2:2.1-53.7.el7_7.x86_64): Cannot allocate memory history info
[root@vps2 ~]#
I may sound old school but my suggestion would be to forget yum aka dnf for now and just fix the real stuff using rpm.
I always considered yum a wrapper for rpm (I wish it was) and was able to fix a system with rpm alone and later make yum behave again.
It usually boils down to analyzing what is in the rpm db and what is on the filesystem actually. You can then remove invalid rpm db content as well as wrong filesystem content. It’s a bit of work and you may create helper scripts as needed but you can fix almost everything that way.
Regards, Simon
Before you try the update again, you’ll have to fix the reason for the failure – add memory, or at least add a swap file.
You could check with rpm -qa –last | head -20
which the latest packages are that were installed.
If the rpm database is corrupted, rebuild it with rpm –rebuilddb
You can reinstall packages that may not be completely installed, using rpm –reinstall PACKAGE_FILE
I’d rpm –erase the kernel package that does not boot.
HTH, Kay
Well, there may be more to check and above steps may not help.
Do you have duplicates in rpmdb? Which one of the duplicates are already on the filesystem?
Regards, Simon
I do not believe that I have any duplicate RPM’s installed, and I do not believe that the database is actually wrong. I believe that it is purely that the installs did not complete successfully.
I have managed to remove the latest kernel RPM and ‘yum update kernel*’ re-applied the update without errors.
I am going to go through the list of failed RPM’s and to a ‘rpm –reinstall’ for each one in turn and hopefully they will work too.
Before doing that I will do as Kay suggested and find out why I had the memory problem in the first place.
If you do not want to involve the ISP ever again when a kernel won’t boot, you can configure a fallback kernel [0]. I’ve used something similar a very long time ago when the main boot loader was LILO – it was much easier to do this kind of thing with it.
[0]: https://anindya.me/2011/09/17/grub-fallback-after-kernel-panic/
went hunting.
If you have a RHEL subscription (I’m using the free developer subscription) you can see the official Red Hat documentation at [0]. I
was unable to find similar instructions for CentOS but here’s a post of the same vintage as the RHEL one [1].
[0]: https://access.redhat.com/solutions/1567883
[1]: https://forums.CentOS.org/viewtopic.php?t=60220
PS: I haven’t tested either of these so no idea if they work as one would expect.