16 thoughts on - Kernel Live Patching On CentOS Stream 9

• There are 2 things to note here about kernel live patching:
  – We do not provide patch files in CentOS Stream (or previously in CentOS Linux, for that matter). We’ve always recommended RHEL as a better fit for folks that have hard requirements on this sort of workflow.
  – RHEL 9 is not yet released, my understanding is that patch files for RHEL proper will begin showing up after GA

  –Brian

• https://access.redhat.com/solutions/2206511

  My understanding of live kernel patching is that the feature allows systems to update specific individual kernel functions, and is primarily useful for addressing security vulnerabilities (and not, for example, for updating from one kernel version to another).  I don’t know for a fact, but my expectation is that CentOS Stream systems aren’t going to get “live” patches because there’s no ongoing support for individual kernels.

• If Stream is to be the next RHEL, wouldn’t you want to test this kind of thing so the RHEL subscribers don’t have to?

• RHEL tests live patches for released, supported kernels before they are delivered to customers.

  josh

• Red Hat does not rely on end-users to test their software.  (And that’s definitely not what Stream is for.)

• Il 2022-01-07 19:07 Brian Stinson ha scritto:

  Yup, this matches my expectations. Thank you all for sharing. Regards.

• Indeed .. you will get those things rolled into the next kernel, but not as live patches.

• In layman’s language summary: RedHat Enterprise features (including
  “live” kernel patching) are to be expected _only_ in RedHat Enterprise
  “binary replica” distributions, which CentOS Stream is not.

  Valeri

• I don’t think that’s true, exactly.  As far as I know, rebuild distributions never had the “Enterprise” features*.  Critically, I think that a lot of people mistakenly believed that CentOS *did* have Enterprise features, because it was rebuilt from RHEL code, and that misunderstanding underlies a great deal of the negative response toward CentOS Stream.

  *: “Enterprise” features include but are not limited to:

  1. Minor releases with independent life cycles / Extended Update Support
  2. Classification for updates (security, bugfix, enhancement)
  3. Live patching for kernel security vulnerabilities
  4. Support

• Thanks for correcting my layman’s representation. It should have better said that “binary replica” is “binary compatible” in a sense whatever software distributed as binary for RHEL will work the same on “binary replica”. I guess my views and wordings got skewed by latest changes of CentOS paradigms.

  We never had it in CentOS in the past, but I’m just curious: is live patching proprietary piece of RHEL? I know there are several solutions, way back there was paid one called splice, my Boss’s son was one of the developers of that. Just curious, as, if it is paid, it is stripped off as part of CentOS composition, but if it is not paid, open source, then it would “just work”, or not?

  Oops, as features I meant functionality of CentOS, nothing beyond that.

  Valeri

• –Indeed, we’re talking the software versus the organization. I never expected CentOS the organization to provide anything more than repackaging
  (rebuilding and mirroring).

  For kernel patching, there’s the matter of rebuilding and distributing the patches, and then whether the software can do anything with that. If it’s proprietary, the issue is moot.

  But maybe it’s like the update classification and differentiation, which was never implemented for CentOS, because of the extra effort the organization would have to provide.

• RHEL’s kernel live patching uses upstream open source kpatch. The sources to the kpatches are delivered in customer facing CDN repos at the same time as the kpatch itself. We do not use proprietary code to produce or apply the kpatches.

  I can only speculate on whether RHEL kpatches would work on a CentOS
  kernel, but my assumption is that they would not due to how they are signed.

  josh

• Il 2022-01-14 13:17 Josh Boyer ha scritto:

  Is (well, was) the CentOS kernel identical at binary level to the RHEL
  one?
  If so, the same kpatch should be applicable to both RHEL and CentOS (the old one).

  But I seem to understand that the two kernels are *not* bytewise identical, so a binary kpatch can not be applied the CentOS. Is this true?

  Anyway, RH kpatches are surely not compatible with CentOS stream. So I
  asked if some project was started to provide live kernel patching to the new CentOS project. If I don’t miss something, this is not the case.

  Regards.

• No .. none of the CentOS Kernels were EVER binary compatible with any RHEL kernel.

  CentOS Linux has always been (now also including CentOS Stream 8 and 9)
  a completely separate ‘closed’ build system.

  We use the SAME source code to build things, modified to remove branding. But CentOS has NEVER been (nor is any other rebuild distribution now) Binary Compatible.

  Want to see how .. just extract two rpms with the same name from two different distributions into separate directories and run a sha256sum on all the files in the different directories with find command. Some files may be identical (most text files that are copied), others will not be.

  It is virtually impossible for all produced packages to be ‘binary compatible’ UNLESS they are built with exact the same files (not files BUILT fromt he same sources .. the exact same files) in the build root AND with exactly the same software doing the building. Any group that claims ‘binary compatibility’ is either lying or they do not understand compiling and linking.

  CentOS never had that. Neither does any rebuild.

  This is why the CentOS Project ‘CHANGED’ our term from binary compatible to ‘Functionally Compatible’ a long time ago. (Using same source code, we produce DIFFERENT software .. that works the same way but has different SHASUM values. Don’t be fooled by key words like ‘binary compatible’ .. check it out for yourself.

  If you build kpatches to kernels, to make them work you need to build the kpatch for the specific kernel (CentOS would need to build against CentOS kernels, etc). Also, there are the certificate signing issues and keys that you would need to take into account. You need to have the CA Trust to be able to create signatures that the system will allow.

• As a bit of a helper .. we used to use this script to find differences:

  https://vault.CentOS.org/4.9/build/distro/tmverifyrpms

  Way back in the CentOS Linux 4 days.

• Il 2022-01-14 15:30 Johnny Hughes ha scritto:

  Thank you so much for the detailed explanation, very appreciated. Regards.