CentOS 6.4 Kernel Panic On Boot After Upgrading Kernel To 2.6.32-431.29.2

Home » CentOS » CentOS 6.4 Kernel Panic On Boot After Upgrading Kernel To 2.6.32-431.29.2
CentOS 16 Comments

I’m on a Supermicro server, X9DA7 motherboard, Intel C602 chipset, 2x 2.4GHz Intel Xeon E5-2665 8-core CPU, 96GB RAM, and I’m running CentOS 6.4.

I just tried to use yum to upgrade the kernel from 2.6.32-358 to
2.6.32-431.29.2. However, I get a kernel panic on boot. The first kernel panic I
got included stuff about acpi, so I tried adding noacpi noapic to the kernel boot parameters, which at least changed the kernel panic message, now I get
(transcribed from a photo I took, so please excuse any errors):

Kernel panic – not syncing: Attempted to kill init!
Pid: 1, comm: init Not tainted 2.6.32-431.29.2.el6.x86_64 #1
Call Trace:
(excluding addresses, please let me know if they’re good for anything)
panic do_exit fput do_group sys_exit_group system_call_fastpath

My grub.conf entry looks like this currently:

title CentOS (2.6.32-431.29.2.el6.x86_64)
root (hd0,0)
kernel /boot/vmlinuz-2.6.32-431.29.2.el6.x86_64 ro root=UUID=ca1e1248-0a65-4b6c-9f87-0c859eab1f17 rd_NO_LUKS rd_NO_LVM
LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto rhgb quiet nomodeset rdblacklist=nouveau KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM verbose selinux=0 enforcing=0 noacpi noapic nolapic nouveau.modeset=0
initrd /boot/initramfs-2.6.32-431.29.2.el6.x86_64.img

(With all the options I added to see if it would work, including a couple from a very similar box that runs fine with that kernel, but on CentOS 6.5.)

This box has a bunch of cards in it, including some NVidia GPUs in a PCI
expander, another NVidia GPU for the GUI, an Areca controller, a Mellanox IB
adapter, and some other stuff.

But, I’m pretty sure this must be something simple I’m missing. Ideas for figuring it out?

16 thoughts on - CentOS 6.4 Kernel Panic On Boot After Upgrading Kernel To 2.6.32-431.29.2

  • Is that a 6.4 kernel? Seems like it ought to be 6.5 from the date.

    Yeah: don’t run random combinations of rpms and then ask the mailing list for support.

  • If yum/rpm allowed him to just upgrade the core kernel witouh the whole system, that means it should be possible to run with it. Please, be positive.

  • I don’t know. yum info doesn’t say anything about it being a 6.4 or 6.5 kernel
    (nor does it say that about the 2.6.32-358 package). How can I tell if a package is intended for 6.4 or 6.5? The release field simply contains “el6”, nothing about minor version.

  • Ok, so is that a confirmation that installing this kernel, even though it might be “for 6.5” should not in itself break anything, and that it should boot?

  • Joakim Ziegler a écrit :

    Every RH errata contains the following text:
    « Before applying this update, make sure all previously released errata relevant to your system have been applied. »

    And that’s the case for that kernel:
    https://rhn.redhat.com/errata/RHSA-2014-1167.html

    So, imho, just yum update and reboot. You’ll be at 6.5 and far more safer.

    My 0.02€. Laurent.

  • Yeah, I might end up doing that. However, this is a system that runs proprietary software that explicitly runs on 6.4, so I was trying to avoid upgrading everything.

    Oh well. It won’t be the first time I make this particular propritary system run on a distro it doesn’t officially support, they were stuck on CentOS 5 for the longest time, and I made it work on 6.3 before they got with the times.

    —
    Joakim Ziegler – Supervisor de postproducción – Terminal joakim@terminalmx.com – 044 55 2971 8514 – 5264 0864

  • I have been doing just that for years with CentOS 5 and CentOS 6 and never, ever had a problem on a whole bunch of different hardware!

  • The advise to do a full upgrade is the best (most secure) option .. however, theoretically, the new kernel should boot and not cause issues based on the other packages.

    I would search for kernel issues for your model server and the 6.5
    kernels, as I think the issue is with some hardware drivers and not related to the other userland packages (unless you happen to be using a kmod somewhere).

    That being said, you should always try to stay current with all updates to maximize the chances that everything works together .. and Laurent is absolutely correct, the only tested ‘upstream’ solution (other than their EUS?AUS solutions) is what he said, which is:

    “Before applying this update, make sure all previously released errata relevant to your system have been applied.”

    As far as what kernel is designed for which release … every point release (minor version) will have its own kernel branch associated with it, you can see this at http://vault.CentOS.org/

    For example, 6.0 has:

    kernel-2.6.32-71.el6

    All the other 6.0 kernels are kernel-2.6.32-71.x.y.el6 .. ie, kernel-2.6.32-71.7.1.el6, kernel-2.6.32-71.14.1, kernel-2.6.32-71.18.1.el6, kernel-2.6.32-71.18.2, etc.

    The following is a breakdown of what each starts with:

    6.0: kernel-2.6.32-71
    6.1: kernel-2.6.32-131
    6.2: kernel-2.6.32-220
    6.3: kernel-2.6.32-279
    6.4: kernel-2.6.32-358
    6.5: kernel-2.6.32-431
    6.6: kernel-2.6.32-504

    Thanks, Johnny Hughes

  • OP said he had an InfiniBand card. For a long time it was the case that every kernel point release needed the corresponding IB userspace libraries.

    It’s also worth noting that running with an unusual set of packages means you’re running something that is not well-tested. Most enterprise users like to run with the herd, in the middle of the herd. Mooo. That basically means “yum -y update”, nothing different.
    “Yum let me do it” and “this is wise” are two different things!

    — greg

  • …

    First question: can you boot with the old kernel still (by default CentOS 6 leaves a few old kernels around; I want to say the default is
    3, but it might be 5, I don’t recall, and I don’t have a straight default C6 install to check against right at the moment)?

    Next question: did you also update the updated kernel-firmware package for the updated kernel?

    The first thing I would do is downgrade the kernel and make sure the system is working there; you then will need to very carefully check all your hardware components together that the kernel update should be ok.
    You mention GPU’s; which drivers are you using there? Iterate over all hardware.

    Now, I’m going to sound like a broken record here. If you absolutely positively must stay at a point release for whatever reason (and there are valid reasons for this), then you don’t need to be running CentOS;
    it is simply not supported. You either need to pay up for RHEL6 with EUS, or you need to install ScientificLinux 6 (built from the same sources that CentOS is built from, with a different rebranding); the SL
    team does support getting only critical updates but staying on a particular point release.

  • …

    To be clear EUS only provides critical updates (+some important) and for a limited time. You can expect ~1y extra (6.3 no longer even EUS
    supported and 6.4 ends in feb 2015).

    /Peter

  • I can confirm that a full upgrade to 6.5 fixed the boot problem. I do think it’s maybe a bit too easy to make this mistake, though.

  • Upgrading to 6.5 did fix the problem, and did not (so far) seem to break my proprietary software.

    For reference, I had already updated the kernel-firmware package (that happened automatically), and I could still boot the old kernel, which was how I got around to upgrading to 6.5.

    More than anything, it’s a little annoying that this is such an easy mistake to make, and so hard (it seems) to debug. But well, I won’t be making it again.