10 thoughts on - Kpatch (live Kernel Patching) In CentOS 7.7?

• The functionality should be available, but we don’t provide patches in this way, no.

• What would it take to make this happen? This would be a huge help to those of us running servers. Not to mention it would make the world a more secure place :)

  Is it an upstream issue? No SRPMS available? Etc?

  Just trying to understand. I don’t follow the CentOS-devel list. Has this been discussed there, or elsewhere?

• There is a lot to go into making a correct kpatch. You have to determine that you have a working kpatch (you can have one which works on 1% and corrupts 80% and crashes 19%), you have to determine that the patch fixes the problem (you can build patches which should do the right thing but don’t), and you have to determine that it doesn’t add in some sort of long term corruption of memory/disk/etc. That takes specialized kernel expertise, a large amount of varied hardware to test the patch on, some amount of time, and a very large test suite.

  You can also only live patch a system so many times and in only certain places. There are just some parts of the kernel which have to be rebooted and others you can put in a patch which works but your performance is going to be 25% of what it was before. There are other places that if you patch.. that is it.. try another and you hardlock. As much as some sites like to call it some sort of panacea for never having to reboot again.. it is really meant to be a tourniquet to air chopter the crash victim to a hospital. They may still not make it… you are just giving them a chance.

• The short answer is “a team of kernel engineers, which we don’t have”. Smooge’s overview which I’ve left below is great at explaining some of this:

  It’s quite a bit more work than just SRPM (re) building. This is one of those things where if your workflow requires this functionality rather than the occasional reboot you should really just pay for RHEL. They put far more people and testing behind this feature than the team building CentOS is able to.

  (DISCLAIMER: I work for RH, so that may not sound as true as it is)

• I don’t understand. If RHEL is putting out patches, and CentOS is a recompile of RHEL, hasn’t that “team of kernel engineers ” already done the work?

  I fully realize this is not a panacea for never rebooting again, but if we can patch a critical kernel bug immediately, then schedule less disruptive reboots in a week or three, this would help tremendously.

  I knew someone was going to say that. :) In our case, as I’m sure is the case for many other environments, we are a noncommercial CentOS shop that can’t afford the resources to have a mixed environment, not to mention the RHEL licenses. Not all of the machines I’m thinking of are critical infrastructure. We have many researchers running simulations that take weeks, sometimes months, to finish, and avoiding the occasional forced immediate reboot for a critical kernel bug would help expand Human Knowledge :).

  Anyway, I saw the functionality for live kernel patching in the RHEL 7.7
  release notes, which the CentOS 7.7.1908 release notes pointed to, and assumed (hoped?) that it would be available for us as well. If it won’t ever be provided, then I suggest the CentOS documentation be updated to explicitly state so.

• No. because most of the work on making a patch is after the kernel is compiled and working. Thus even though you have the same source code, similar compilers etc.. there are going to be differences which have to be looked at to make sure it is really working. A CentOS kernel is not exactly the same as a RHEL kernel is not the same as a Oracle kernel is not the same as the one you recompiled locally. From most operational points they seem the same, but kernel patching is where those differences really show up.

  Yes it would be easy to set up some automated tool which ‘made’
  kpatches.. and I expect they may ‘work’ for most systems. But I also expect that they would also eat babies more times than people would like. If sites really need them, they can set up the tooling themselves and make them work when they know they want it. Trying to make it a general purpose answer for something which may corrupt data
  5 or 20% or 40% of the time.. is just waiting to be on Slashdot daily
  (wait do we do Slashdot anymore.. Reddit? nope the kids aren’t there anymore either.. ok someplace daily) in a bad way.

• Thanks for the explanation(s).

  I’m still puzzled why RedHat is doing it then, and making it more generally available (to paying customers even), if it’s so dire a proposition that it will fail so badly, so often. That seems counter-intuitive to me.

  Anyway, I again point out that the CentOS documentation should be made clear that this functionality won’t ever be coming to CentOS.

  -Matt

• Am 04.10.19 um 15:35 schrieb Phelps, Matthew:

  I’ve been using kernel live patching on an Ubuntu Machine for several years now without any problems (Ubuntu offers them for registered users for free (three machines per account)) and haven’t noticed any downsides so far.

• Because they have kernel developers, qa, and other staff dedicated to making that kpatch work? They have a large set of servers to test different workloads? They have some time before the kernel is built internally and when it is made available externally to do all this and hand tune any problems found? Because big companies are paying a large amount of money to make it work and so the extra labour is profitable?

  In the past, all of this would be a challenge for people to come together and show that they can also do it themselves… or improve on something to make it so less labour intensive at parts. If that happens, I am happy to have laid out the challenge :).

• It would likely boil down to a risk-benefit analysis; for RHEL RH is willing to take the risks associated with it due to the added benefits of offering it.