Https://blog.CentOS.org/2020/12/future-is-CentOS-stream/

Home » General » Https://blog.CentOS.org/2020/12/future-is-CentOS-stream/
General 150 Comments

The future of the CentOS Project is CentOS Stream, and over the next year we’ll be shifting focus from CentOS Linux, the rebuild of Red Hat Enterprise Linux (RHEL), to CentOS Stream, which tracks just ahead of a current RHEL release. CentOS Linux 8, as a rebuild of RHEL 8, will end at the end of 2021. CentOS Stream continues after that date, serving as the upstream (development) branch of Red Hat Enterprise Linux.

Meanwhile, we understand many of you are deeply invested in CentOS Linux
7, and we’ll continue to produce that version through the remainder of the RHEL 7 life cycle. https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates

CentOS Stream will also be the centerpiece of a major shift in collaboration among the CentOS Special Interest Groups (SIGs). This ensures SIGs are developing and testing against what becomes the next version of RHEL. This also provides SIGs a clear single goal, rather than having to build and test for two releases. It gives the CentOS
contributor community a great deal of influence in the future of RHEL. And it removes confusion around what “CentOS” means in the Linux distribution ecosystem.

When CentOS Linux 8 (the rebuild of RHEL8) ends, your best option will be to migrate to CentOS Stream 8, which is a small delta from CentOS
Linux 8, and has regular updates like traditional CentOS Linux releases. If you are using CentOS Linux 8 in a production environment, and are concerned that CentOS Stream will not meet your needs, we encourage you to contact Red Hat about options.

We have an FAQ – https://CentOS.org/distro-faq/ – to help with your information and planning needs, as you figure out how this shift of project focus might affect you.

[See also: Red Hat’s perspective on this. https://www.redhat.com/en/blog/CentOS-stream-building-innovative-future-enterprise-linux]

150 thoughts on - Https://blog.CentOS.org/2020/12/future-is-CentOS-stream/

  • Wow, major shift in philosophy, thanks for all the hard work over the years but if I wanted to be a beta tester there are many distributions that will serve that purpose, real shame.

    The king is dead, long live the king!

  • This is really, really bad for the majority of us using CentOS.

    Is there any way we can lobby for the reversal of this decision? Remember that the -devel mailing list, and IRC channels *do not* represent the vast majority of CentOS users. Most of us are just sysadmins trying to keep our systems that have been using CentOS for many, many years running and our procedures for installing, and patching systems working after whatever changes have been mysteriously decided upon, and forced on us.

    We will be forced to look at other distributions now; and forced to do a ton of unnecessary work to deal with this.

    Thanks a lot.

    —

    *Matt Phelps*

    *Information Technology Specialist, Systems Administrator*

    (Computation Facility, Smithsonian Astrophysical Observatory)

    Center for Astrophysics | Harvard & Smithsonian

    60 Garden Street | MS 39 | Cambridge, MA 02138
    email: mphelps@cfa.harvard.edu

    cfa.harvard.edu | Facebook <http://cfa.harvard.edu/facebook> | Twitter
    <http://cfa.harvard.edu/twitter> | YouTube <http://cfa.harvard.edu/youtube>
    | Newsletter <http://cfa.harvard.edu/newsletter>

  • Aside from the the latest shiny – what are the advantages of CentOS 8 Stream? What are the benefits?

    I read through the announcement and FAQ, but they do not address that question. Is it just a name change? Is it an attempt to put CentOS on a subscription model?

    —
    Bill Gee

  • I agree this is shocking news.  If we don’t want to be beta testers and want to continue to use a stable tested OS should we be moving to RHEL
    servers?  Is there a license-free RHEL server option that is the recommended path from using CentOS?

    Chris

    —
    Christopher Wensink IS Administrator Five Star Plastics, Inc
    1339 Continental Drive Eau Claire, WI 54701
    Office: 715-831-1682
    Mobile: 715-563-3112
    Fax: 715-831-6075
    cwensink@five-star-plastics.com http://www.five-star-plastics.com

  • We would also have to spend an extraordinary amount of work moving off CentOS. We currently run it on production servers and would no longer be able to run it if it was beta.
    This will affect thousands of CentOS business users who use it because it is stable and not a beta.

    Andrea

    —–Original Message—

  • LOL, laughing with you not at you, license free RHEL. Just RHEL (IBM)
    wanting to increase the coffers for RHEL. I know, I know thats not whats really happening but yeah that is what is really happening. If you want a tested production worthy server install based on RHEL then you will have to pay for RHEL. I know this is exaggerating but if I wanted to test pre-production stuff I could just run Fedora. ;)

    Oh well end of an era…sniff.

    servers? Is there a license-free RHEL server option that is the CentOS mailing list CentOS@CentOS.org https://lists.CentOS.org/mailman/listinfo/CentOS

  • Of course it is.

    The reality is that it was always on borrowed time.

    Getting RHEL without paying for it and with slight delays in updates (most people don’t even update that often anyway) wasn’t going to be sustainable, ever.

    If your business case resolves around being able to freeload on the work of others, then there’s a serious problem with the business case.

    And I say that as somebody who has installed a large portion of the CentOS8 (and 7) servers at work.

    Not sure what we ourselves are going to do about it, though.

    I would hate to switch to Ubuntu for the stuff I like CentOS most for (for some, it’s arguably not the greatest distro).

    We might end up licensing RHEL for that – and the rest maybe Fedora.

  • I DO NOT WORK FOR A BUSINESS!! (Yes, I’m yelling. Sorry, but I am well and truly pissed off!) We have no budget for RHEL.

    —

    *Matt Phelps*

    *Information Technology Specialist, Systems Administrator*

    (Computation Facility, Smithsonian Astrophysical Observatory)

    Center for Astrophysics | Harvard & Smithsonian

    60 Garden Street | MS 39 | Cambridge, MA 02138
    email: mphelps@cfa.harvard.edu

    cfa.harvard.edu | Facebook <http://cfa.harvard.edu/facebook> | Twitter
    <http://cfa.harvard.edu/twitter> | YouTube <http://cfa.harvard.edu/youtube>
    | Newsletter <http://cfa.harvard.edu/newsletter>

  • Stream is the RHEL sorce code for rhel + 0.1 .. so durng the 8.3 rhel cycle, stream will be rhel 8.4 source code.

    It is not very far ahead of the current code. It is indeed the code you will get in 6 months. It is not ‘new shiny’ .. it is newer enterprise.

    What are the benefits:

    1) Many people (like Intel and Facebook) are providing feedback in real time. So can any user. They should have in place, before RHEL 9
    development starts, the ability to accept public community pull requests into stream.

    2) This code is still RHEL source code .. it is just not released in rhel yet. Almost all of it will be released in the upcoming RHEL point release.

    3) Most bugs will get fixed faster, if the code is pulled into stream. Many times you don’t get the fix until the next point release .. and this will be what stream is.

  • Am 08.12.20 um 15:15 schrieb Tom Bishop:

    Making such a cut at the beginning of CentOS8 life is remarkable. We did a lot work to migrate to C8? Honestly, this had to be done before C8 or at the end of C8 but not now?

    —
    Leon

  • Another very good thing

    There is no longer a huge delay of a drop of 750 packages at once and a delay of more than a month to get a new release.

    There will be on delay in stream .. it will be a constantly rolling distribution .. updates will happen all the time with no ‘big drop’ at point release time.

    Just do a thought experiment and think about the current process and how stream works. There is no reason you can not use CentOS Stream for what you use CentOS Linux for now.

    I know what the knee jerk reaction is .. but put that aside and really look at the timing of code and release of packages, etc.

    If you don’t like how it turns out .. then shift to another platform. You (and we) have a full year to show you the benefits of Stream.

    Is Ubuntu or Debian any better wrt package versions and enterprise?

    How really different is RHEL 8.3 from 8.2 .. or 8.4 from 8.3?

    Just look at the facts and make the choice that is appropriate for your situation.

  • Forgive a bit of cynicism …

    “If you want to keep using RHEL for free, you will have to put up with making sure that our paying customers get better quality releases”

    “If you really want to have a stable release for free, stick to 7”

    “CentOS will become the developer playground”

    Was there any confusion? If there is, then it’s caused by the introduction of things like “CentOS Stream”. There was never any confusion when it was a straight rebuild.

    “If you want a production environment, pay for it”

    The FAQ generally says “if you want a RHEL environment, then pay for it”

    Red Hat’s perspective is “CentOS is ours now; IBM have told us to make sure it’s pulling its weight or we aren’t allowed to put any resources into it”

    So as far as I can see all the RHEL rebuilds are dead now – WhiteBox, Scientific Linux, now CentOS. Are there any left?

    P.

  • It appears that Red Hat has taken ownership of a free and opensource OS and is now making decisions regarding it. Maybe it is time to look at Oracle Linux.

    Andrea

    —–Original Message—

  • You have published a CentOS Lifecycle that states the EOL for CentOS 8 is May 2029. (c.f. https://endoflife.software/operating-systems/linux/CentOS). CentOS Stream *is not* CentOS 8.

    This announcement is a breach of that trust with your community, and could be construed as a breach of contract with your users.

    Save this change for CentOS 9.

    —

    *Matt Phelps*

    *Information Technology Specialist, Systems Administrator*

    (Computation Facility, Smithsonian Astrophysical Observatory)

    Center for Astrophysics | Harvard & Smithsonian

    60 Garden Street | MS 39 | Cambridge, MA 02138
    email: mphelps@cfa.harvard.edu

    cfa.harvard.edu | Facebook <http://cfa.harvard.edu/facebook> | Twitter
    <http://cfa.harvard.edu/twitter> | YouTube <http://cfa.harvard.edu/youtube>
    | Newsletter <http://cfa.harvard.edu/newsletter>

  • This statement from that page is also now a lie:

    The CentOS Linux distribution is a stable, predictable, manageable and reproducible platform derived from the sources of Red Hat Enterprise Linux
    (RHEL).

    This statement is the reason we *all* chose CentOS. You will be betraying us in the worst possible way if this change is allowed to happen.

    This isn’t just a “Knee jerk” reaction. CentOS is abandoning everything it stood for by even creating CentOS Stream. We *dont want it* !!

    Getting rid of a strict re-compile of RHEL X.X is a complete reversal of the principles of CentOS and the community it serves. Again, WE DON’T WANT
    THIS!

    —

    *Matt Phelps*

    *Information Technology Specialist, Systems Administrator*

    (Computation Facility, Smithsonian Astrophysical Observatory)

    Center for Astrophysics | Harvard & Smithsonian

    60 Garden Street | MS 39 | Cambridge, MA 02138
    email: mphelps@cfa.harvard.edu

    cfa.harvard.edu | Facebook <http://cfa.harvard.edu/facebook> | Twitter
    <http://cfa.harvard.edu/twitter> | YouTube <http://cfa.harvard.edu/youtube>
    | Newsletter <http://cfa.harvard.edu/newsletter>

  • This really pisses me off. You published CentOS 8 with a promise to support it until May 2029.

    Now you betray all users that took you by the mouth by stating it’s EOL december 31. 2021.

    Do you really think that was a smart move?

  • I am already using Oracle Linux in some servers.

    Free as CentOS, faster updates than CentOS, and with some extra support, BTRFS and a newer kernel, for example.

  • Many CentOS users are also RedHat customers. Those RedHat customers may currently be rethinking their RedHat investment given this change of events. We currently run Oracle Linux on a number of servers and have been very pleased with it.

    Andrea

    —–Original Message—

  • So did we. It’s in Git.

    That page was never a contract. It’s a web page published by an open source project. Please do not misconstrue it as a contract.

  • Of course it was not a legally binding contract. But we, the users, trusted you. No we don’t anymore.

  • Hi,

    this is really bad news.

    Back in 2014 [1], sadly no one at RH seems to remember…

    “Some of the things that are not changing:
    – – The CentOS Linux platform isn’t changing. The process and methods built up around the platform however are going to become more open, more inclusive and transparent.
    – – The sponsor driven content network that has been central to the success of the CentOS efforts over the years stays intact.
    – – The bugs, issues, and incident handling process stays as it has been with more opportunities for community members to get involved at various stages of the process.
    – – The Red Hat Enterprise Linux to CentOS firewall will also remain. Members and contributors to the CentOS efforts are still isolated from the RHEL Groups inside Red Hat, with the only interface being srpm /
    source path tracking, no sooner than is considered released. In summary: we retain an upstream.

    Feel free to reach out if you have specific concerns about how this change impacts your CentOS story. URLs mentioned at the bottom of this email should be a good starting point.”

    Crossing fingers that alternatives emerge soon.

    Best regards, Markus

    [1]
    https://lists.CentOS.org/pipermail/CentOS-announce/2014-January/020100.html

  • I don’t think anyone seriously thought it was a contract.

    Open Source works largely on trust. Trust that the developers aren’t going to intentionally harm their users, and the trust that those developers will provide a consistent product. Developers earn the trust of their users. Trust is the basic commodity for Open Source.

    That doesn’t mean that open source providers always provide those things. There are many, many stories out there where upstream makes an abrupt change that their users dislike. And if you’ve broken that trust, end users are going to be wary of ever putting any more trust in the developers.

    Maybe the CentOS Stream thing will work out OK for everyone. But the way this was announced, there are a lot of people who have lost trust in CentOS and Red Hat. Changing the end of life for CentOS 8 has broken our trust in the project.

    I see a lot of promises that Stream will have better engagement with the community. Why would we trust these promises?

  • To be fair, that was a commitment RH gave. They are now a department within Big Blue and must dance to their tune. Of course you could always try holding your breath and awaiting the sell off to Lenovo in about five years time.

  • Maybe not a contract, but clearly CentOS set that timeline in the past and has now abandoned it, on extremely short notice, and that is a major breach of trust.

    Changing a product timeline BY EIGHT YEARS is a huge, huge change, and it completely erodes any confidence your community has in you.

    I respectfully request this change be postponed until CentOS 9 and announced as such from the beginning.

  • Le 08/12/2020 à 16:12, Johnny Hughes a écrit :

    When I read the first messages of this thread, I was quite concerned. But having read through your detailed explanations, let me state that I’m reassured.

    As a sysadmin, what I like about CentOS is that it’s probably the most *boring*
    Linux distribution out there. Boring is good. No drama, no surprises. I know I
    can have yum-cron run once a day without Icinga suddenly sending me a tsunami of failure alerts and without clients calling me and yelling on the phone.

    Cheers,

    Niki

    —
    Microlinux – Solutions informatiques durables
    7, place de l’église – 30730 Montpezat Site : https://www.microlinux.fr Blog : https://blog.microlinux.fr Mail : info@microlinux.fr Tél. : 04 66 63 10 32
    Mob. : 06 51 80 12 12

  • “Better engagement with the community” of course requires that the community step up and engage. Trust isn’t something anyone expects to just magically happen.

  • The whole point of CentOS was so that we didn’t have to “engage.” We don’t have time for that.

    We just want a stable re-compile of RHEL, as promised. CentOS has been diverging from this for a while (note the change in version names/numbers)
    and we DON’T WANT THAT!

  • I still haven’t seen an answer to the question, “Who made this decision?”
    and, “How can we lobby to get it changed?”

    —

    *Matt Phelps*

    *Information Technology Specialist, Systems Administrator*

    (Computation Facility, Smithsonian Astrophysical Observatory)

    Center for Astrophysics | Harvard & Smithsonian

    60 Garden Street | MS 39 | Cambridge, MA 02138
    email: mphelps@cfa.harvard.edu

    cfa.harvard.edu | Facebook <http://cfa.harvard.edu/facebook> | Twitter
    <http://cfa.harvard.edu/twitter> | YouTube <http://cfa.harvard.edu/youtube>
    | Newsletter <http://cfa.harvard.edu/newsletter>

  • You do understand that Open Source does not work like that?

    If you cannot justify the expenses for RHEL, then you need to compromise. That’s like requesting free Windows licenses.

    Either use Fedora, or CentOS Stream or something different.

    You will likely find, however, that most Open Source software is driven by the people who commit code (the successful ones at least).

    Those who commit code are nowadays usually employed by a company, which in itself either makes money directly or indirectly from the work of the people who commit the code.

    So, you will quickly be back to square one, unless you want to run stuff like Debian or Ubuntu, which are mainly Linux-kernel+some stuff nowadays, whereas RHEL + CentOS forms a complete system (with additional software that RedHat has developed or acquired over the years).

    Debian + Ubuntu are no replacements for CentOS/RHEL, IMO. They are something different.

  • I want what was promised to us in the past. On the schedule that was already published.

    Here is how you can convert:

    No, it’s *because* of you (collective “you,” that is) that we have to do way more work than we planned, or have budgeted for. It’s not “up to me.”
    There are many other forces here that are extremely hard to change. I’m not talking about reinstalling my laptop. Remember that the “E” stands for
    “Enterprise.”

    Can you not see that we are pissed off because what was advertised as being stable through 2029 has been swept out from under us? All you have to do is postpone this *massive* change to CentOS 9, and announce that from the beginning.

    The reaction to this is universally negative.

  • Clearly, you haven’t been around here long. :)

    This is par for the course here, and extremely frustrating.

    —

    *Matt Phelps*

    *Information Technology Specialist, Systems Administrator*

    (Computation Facility, Smithsonian Astrophysical Observatory)

    Center for Astrophysics | Harvard & Smithsonian

    60 Garden Street | MS 39 | Cambridge, MA 02138
    email: mphelps@cfa.harvard.edu

    cfa.harvard.edu | Facebook <http://cfa.harvard.edu/facebook> | Twitter
    <http://cfa.harvard.edu/twitter> | YouTube <http://cfa.harvard.edu/youtube>
    | Newsletter <http://cfa.harvard.edu/newsletter>

  • The first thing Oracle wants is for you to sign up for an Oracle account. Hmm, I’ll give Springdale a try. For those with long memories, remember the DEC RDMS promises prior to take over, and the aftermath?

  • I am sorry you feel that way. I was trying to help. Red Hat has several programs where they give away RHEL for free. CentOS Stream is free. CentOS Linux 8 is going away in at the end of 2021.

    Those are facts.

    He said he wanted free Linux and did not want to interact with the community. I provided all the alternatives that i know for that.

    Stream, a qualified version of RHEL that can be obtained from Red Hat for free (has nothing to do with CentOS) or some other distribution.

    I am not being arrogant or negative. Just pointing out facts.

    I have been doing this for 17 years. I would continue doing for 17
    more. But it is what it is and wishing for it to be different is not going to happen. I know .. I’ve tried.

  • That doesn’t sound much like a “community” to me. That’s my point.

    (And I explicitly did not ask for “free Linux,” or even “free RHEL.” I only asked for what the CentOS web page says, and has said for your 17 years.)

    —

    *Matt Phelps*

    *Information Technology Specialist, Systems Administrator*

    (Computation Facility, Smithsonian Astrophysical Observatory)

    Center for Astrophysics | Harvard & Smithsonian

    60 Garden Street | MS 39 | Cambridge, MA 02138
    email: mphelps@cfa.harvard.edu

    cfa.harvard.edu | Facebook <http://cfa.harvard.edu/facebook> | Twitter
    <http://cfa.harvard.edu/twitter> | YouTube <http://cfa.harvard.edu/youtube>
    | Newsletter <http://cfa.harvard.edu/newsletter>

  • “Since March 2004, CentOS Linux has been a community-supported distribution derived from sources freely provided to the public by Red Hat. As such, CentOS Linux aims to be functionally compatible with RHEL.”

    —

    *Matt Phelps*

    *Information Technology Specialist, Systems Administrator*

    (Computation Facility, Smithsonian Astrophysical Observatory)

    Center for Astrophysics | Harvard & Smithsonian

    60 Garden Street | MS 39 | Cambridge, MA 02138
    email: mphelps@cfa.harvard.edu

    cfa.harvard.edu | Facebook <http://cfa.harvard.edu/facebook> | Twitter
    <http://cfa.harvard.edu/twitter> | YouTube <http://cfa.harvard.edu/youtube>
    | Newsletter <http://cfa.harvard.edu/newsletter>

  • I hate to jump in as a Devil’s Advocate here, but that says CentOS Linux. CentOS Linux is being discontinued, so that statement no longer applies to the project.

  • We owe everyone who worked on CentOS a big thank you.

    I think a lot of people are overwhelmed by the fact that the CentOS we knew appears to be dying (was killed, in fact).

    I wonder what the ultimate outcome will be. Probably RHEL will get a few new subscribers and some CentOS users will migrate to stream, but I
    think this will ultimately diminish Red Hat within the Linux world. 
    Probably net advantage to Ubuntu.

    I shudder to imagine a world where Oracle Linux replaces CentOS.

    -Alan

    —

    Alan D. Mead, Ph.D. President, Talent Algorithms Inc.

    science + technology = better workers

    http://www.alanmead.org

    The irony of this … is that the Internet is both almost-infinitely expandable, while at the same time constrained within its own pre-defined box. And if that makes no sense to you, just reflect on the existence of Facebook. We have the vastness of the internet and yet billions of people decided to spend most of them time within a horribly designed, fake-news emporium of a website that sucks every possible piece of personal information out of you so it can sell it to others. And they see nothing wrong with that.

    — Kieren McCarthy, commenting on why we are not
    all using IPv6

  • All,

    Please sign this petition if you don’t want the CentOS Board to implement this decision:

    https://www.change.org/p/CentOS-governing-board-do-not-destroy-CentOS-by-using-it-as-a-rhel-upstream

    Or… don’t if you are happy with the change.

    —

    *Matt Phelps*

    *Information Technology Specialist, Systems Administrator*

    (Computation Facility, Smithsonian Astrophysical Observatory)

    Center for Astrophysics | Harvard & Smithsonian

    60 Garden Street | MS 39 | Cambridge, MA 02138
    email: mphelps@cfa.harvard.edu

    cfa.harvard.edu | Facebook <http://cfa.harvard.edu/facebook> | Twitter
    <http://cfa.harvard.edu/twitter> | YouTube <http://cfa.harvard.edu/youtube>
    | Newsletter <http://cfa.harvard.edu/newsletter>

  • I suppose I understand the negative feedback — CentOS 8.x will no longer be a rebuild of RHEL 8.x but will instead be some version of RHEL 8.(x + 1) — but I’m much more interested in empirical results than in suppositions. I’ve taken a couple test VMs and set them to CentOS 8 Stream and will keep an eye on them. They will either prove stable or not, but (observation > guessing) in my book.

    If history is any guide, they will prove very stable. If not, then I’ll pour one out for CentOS and look elsewhere.

    —
    Paul Heinlein heinlein@madboa.com
    45°38′ N, 122°6′ W

  • If anyone is considering to fork CentOS 8 (I’m not talking about that
    “Stream”), count me in.

    Otherwise I will switch to openSUSE Leap. At least they are not pushing me some testing ground.

    Best Regards, Strahil Nikolov

    В 12:07 -0500 на 08.12.2020 (вт), Phelps, Matthew написа:

  • You are putting lipstick on a pig. Let’s face it: This is IBM pulling the plug on CentOS.

    Not a single one of those “benefits” will benefit *me*. I am a private user hosting his own machines with CentOS for stability but using RHEL
    for work. I do not have the money to pay for RHEL. But I do contribute to open-source projects, some of which are part of RHEL.

    I’m pretty sure IBM is behind this: They still do not like the open-source model. They only like money.

    After 20 years of running and advocating for Redhat based Distros
    (Fedora on workstations, CentOS on servers) I night have to jump ship
    (if somebody is going to clone “classic” CentOS to keep tracing RHEL I
    might reconsider). Debian or Ubuntu: here I come. I will also no longer advocate for RHEL in the workplace where we used CentOS for non-production machines and RHEL for production.

    Thanks for the hard work you put into CentOS over the years. Sorry to hear that it now turns out to have been wasted.

    peter

  • Which is the approach I recommend everyone take.

    And, it will likely be sometime mid to late 1st quarter 2021 before CentOS Stream is in its ‘Fully Functional’ state with community pull requests and the RHEL package maintainer doing all the work in CentOS
    Stream, etc . CentOS Linux 8 will still be available and updated until the end of December 2021.

  • I promise you, to the best of my knowledge, IBM had nothing to do with this decision. Red Hat is a distinct unit inside IBM and Red Hat still has a CEO, CFO, etc. Red Hat also maintains a neutral relationship with many IBM competitors. So this was not an IBM decision.

  • So why is the hurry ? Why this was not done when the EL8 came alive ?

    Best Regards, Strahil Nikolov

  • Le 08/12/2020 à 21:56, Johnny Hughes a écrit :

    I’ve spent the last couple hours reading through various reactions to this sudden paradigm shift, and they’re overwhelmingly negative. Even the brazen professionals and the hardcore guru admins who have seen it all add a little
    “RIP CentOS” to their tweets, blog articles and other publications.

    Only last month I held my yearly 101 class about Linux and Open Source at our local university here in South France. We were talking about enterprise class Linux – which isn’t necessarily commercial Linux – and I remember explaining to my students the choice of CentOS and the benefits of low-risk updates over an extended period of ten years.

    A colleague of mine – the most proficient admin I personally know – already decided to move to Oracle Linux. And I’m currently considering it as an option.

    Cheers from the sunny South of France,

    Niki

    —
    Microlinux – Solutions informatiques durables
    7, place de l’église – 30730 Montpezat Site : https://www.microlinux.fr Blog : https://blog.microlinux.fr Mail : info@microlinux.fr Tél. : 04 66 63 10 32
    Mob. : 06 51 80 12 12

  • But, was this a RedHat decision? In other words, was the CentOS Board influenced by RedHat to make this decision in an effort to generate more revenue by forcing users to switch to a RHEL paid subscription to keep the status quo?

    If so, I assure them, based on all the feedback I’ve seen so far, this decision will backfire.

  • This sounds like a reasonable path forward, but I wonder if Oracle will simply follow CentOS into the same “stream” thing.

    Prior to this point it’s been a difference without any difference, but I wonder if Oracle actually re-creates RHEL or if they re-create CentOS.

  • Then WHO made the decision?

    Where was the transparency in this decision by the CentOS Board?
    (assuming CentOS still *has* a working independent board)

    Judging from the reactions, I don’t believe that anyone saw this coming. Where was the community consultation on IF this was a good idea.

    Jim

  • And Larry Ellison is sitting in his office laughing his ass off. Happy guy with a nice christmas gift from the RH board of directors.

  • So: other than the developer subscription, not yet. But see this part of the FAQ — https://www.redhat.com/en/blog/faq-CentOS-stream-updates#Q10:

    In the first half of 2021, we will be introducing low- or no-cost
    programs for a variety of use cases, including options for open
    source projects and communities, partner ecosystems and an
    expansion of the use cases of the Red Hat Enterprise Linux
    Developer subscription to better serve the needs of systems
    administrators and partner developers. We’ll share more details on
    these initiatives as they become available.

    I’m not part of any of the decisions around that, but I know for sure that work is in progress and it’s not just meant to sound nice. I don’t know if any of these will meet your use cases, but I think they will for a lot of people here.

    For others, note that the plan is for CentOS Stream to target upcoming RHEL minor releases. Between any two six months, the change delta should be just the same as it is in CentOS Linux now. It’s not like it’s going to become Fedora Rawhide. Everything going into it is intended to land in RHEL on a short timescale. It’s not a beta or a playground for broken code.

    Is it possible that more regressions will get through than have before?
    Well, sure, some. But let’s not pretend that even RHEL is ever regression-free. It’s software, after all, and there are bugs and errata. I don’t think that for most self-supported CentOS use, it will be particularly dangerous to switch to Stream at all.

    And if your use case isn’t covered by one of the upcoming low- and no-cost programs, and you can’t take the risk or the possible increased change management overhead, or for some other reason… well, is it
    _really_ so bad for companies to pay for RHEL? (I like my family to be able to eat, so I’m a bit biased…. but all of this has to come from something.)

    —
    Matthew Miller

    Fedora Project Leader

  • Sure, some cynicism is absolutely warranted. It’s a big change.

    I mean…. That’s not the _worst_ deal I’ve ever heard. But actually it’s better than you’ve stated, because the benefit to others happens even regardless of the paying customers — others using CentOS Stream also benefit. And for that matter since many of fixes go upstream, users of open source in general.

    (In some ways, this is like: being a paying customer of RHEL also benefits other paying customers. And for that matter, those paying customers benefit all of the free users, and Fedora, and hundreds of upstreams.)

    This one is categorically not the case. Even Fedora isn’t a developer playground. Everything landing in CentOS Stream is actually *planned* (with emphasis intentional) to go in a future RHEL release.

    Previously, all the development around RHEL releases was done in secret, in the Red Hat black box. Now it’s out of the box and can be watched. There may be some launch pains, but I expect the average quality of an update hitting CentOS Stream to be very high.

    Of course, that link says nothing of the sort. It’s easy to imagine IBM
    conspiracies, but the honest truth is that there’s nothing to that. Now, I
    don’t know everything, and it may be the case that IBM is secretly pulling all sorts of invisible strings and making Red Hat management dance, but I do know about *this* particular thing and IBM had nothing to do with it.

  • Yeah, um, if I’m coming across as arrogant, I apologize. I genuinely think this is overall a good thing, and I’m hoping some of y’all will see some of that. I realize my excitement might not be completely catching at first.

    But Johnny is easily in the running for “least arrogant person I know”, and has demonstrated that consistantly for years and years, as anyone who has followed CentOS regularly will attest. So, c’mon, let’s please not take this there.

  • Forget about IBM.

    We were told way back in 2014, when Red Hat bought CentOS, that CentOS
    would remain independent from RedHat. This is clearly bullshit now. So we were totally misled into adopting the platform with the belief that what we were getting would stay the same, an unsupported recompile of RHEL and its updates.

    This is no longer what we will be getting, and even worse, it is no longer what we will be getting AFTER BEING ASSURED that CentOS 8 would remain in parallel with RHEL 8 until 2029.

    This is a classic bait-and-switch money grab and it is going to hurt *a lot* of people in the form of a shitload of added work to change platforms.

    It’s totally contemptible and a complete betrayal to loyal enthusiasts.

    I have ZERO respect for RedHat now, and unless the decision is reversed, I’ll prosthelytize against them as loud as I can, on as many platforms as I
    can.

    I . Am. PISSED.

    -Matt

    P.S. Where is Karanbir? He is supposedly the Chair of the CentOS Board. I
    want to hear from him on this.

  • The CentOS Board has failed in its *FIRST* four responsibilities.

    From that first web page:

    CentOS Governing Board Responsibilities

    – Guidance and leadership over the ultimate Project roadmap.
    – Community outreach.
    – Maintenance of health and viability of CentOS community.
    – Maintenance of a healthy and proactive relationship with the Project
    users and consider those needs and uses in decisions.

  • B. The Liaison may, in exceptional circumstances, make a decision on behalf of the Board if a consensus has not been reached on an issue that is deemed time or business critical by Red Hat.

    So, Johnny, are you saying that the RedHat Liaison, who is Brian Exelbierd (
    bexelbie@redhat.com) has forced this decision on the CentOS Board, despite objection from the Board?

    Brian, please answer this directly, or the so called “Transparency” from RedHat and CentOS we were promised will be clearly shown to be a lie.

  • I too would be interested to know what happens to CentOS 8 Stream once focus of RedHat moves to RHEL 9? The life cycle document says the last release of RHEL8 will be 8.10, that’s a five year road map (since point releases seem to be every 6 months), so the point releases will end in
    2024, presumably the end of the point releases means the end of Stream updates? Have these things been thought out that far ahead?

    P.

  • As I understand it, the current plan is for there to be Stream 8 and Stream
    9 in parallel. For one thing, once RH developers are all geared up for working in Stream 8, it’d be _extra_ work to pull that all back in-house.

  • For the record, I don’t think this is a good decision because it changes what CentOS is (its “core mission” in business-speak).

  • It’s all the talk of SIGs and developing and testing and that Stream will be the centerpiece of that. That’s what I meant.

    I don’t get that from the documents released today. If Stream is *not*
    a test-bed, then surely the code that appears in Stream must be fully formed in secret behind the scenes first. Yes, it will appear piecemeal rather than in one big chunk, but it has been categorically denied that Stream is not a RHEL 8.n+1 beta and is more a RHEL 8.n+1 RC/rolling release.

    I think what a lot of people are concerned about is the rolling-release aspect of this. There will be no definitive versioning of CentOS in the future – all you will be able to say is “fully updated” and it won’t be possible to slot a CentOS system in to exactly match a RHEL version. Will third party RPMs built against RHEL 8.x be installable on a CentOS
    8 Stream system? The answer is surely “it depends”, but there are a lot of hardware vendors that target drivers to RHEL releases, which may well make CentOS non-viable for hardware that doesn’t have drivers built in to the kernel.

    I suspect that for a large proportion of scenarios Streams will be perfectly OK. But we still get software/instruments that specifically say “only RHEL 7.4” or something like that (yes, it’s a support nightmare).

    P.

  • Was this decision forced, despite objection, on the CentOS board by the RedHat Liaison?

    Yes or No?

    —

    *Matt Phelps*

    *Information Technology Specialist, Systems Administrator*

    (Computation Facility, Smithsonian Astrophysical Observatory)

    Center for Astrophysics | Harvard & Smithsonian

    60 Garden Street | MS 39 | Cambridge, MA 02138
    email: mphelps@cfa.harvard.edu

    cfa.harvard.edu | Facebook <http://cfa.harvard.edu/facebook> | Twitter
    <http://cfa.harvard.edu/twitter> | YouTube <http://cfa.harvard.edu/youtube>
    | Newsletter <http://cfa.harvard.edu/newsletter>

  • No .. I am saying that the CentOS Board knows that option exists, and therefore we know that IF it is invoked, we get no say in what will happen for that decision. I am saying that the CentOS Board therefore made a hard decision given the situation we were in.

  • It is not the same as Rawhide is all I am saying.

    It is based on the current release and it is being modified for some reason.

    That modification can be a bugfix from a reported bug, it can be an enhancement for a given package or it can be a security update.

    Each of these updates will be rolled in one at a time.

    It is what will eventually become the next rhel source code in a few months for the next point release.

    Only you will know if this will work for your situation.

    We do have CI testing, which will be beefed up to be similar to the testing RHEL actually does right now before release of packages for the
    ‘released version’ of RHEL.

    Will bugs get though .. sure, they do now into RHEL.

    But I will absolutely say that the things they are rolling into RHEL 8.4
    in a few months are not inherently less stable or less secure or whatever else you want to call it .. when compared to other Linux distros.

    The process that Debian would use to roll in bug fixes or Ubuntu would use to roll in bug fixes would not be significantly better or more stable or more secure.

  • Then I go back to my previous message, the Board failed to properly engage, and represent the Project’s users.

    Sorry, but basically saying you caved to RedHat’s pressure is worse to me than if you forced them to invoke this option.

    It’s clear from our reaction here, and *everywhere else* that this decision is *not* in the best interest of CentOS’s users.

    You might as well all resign from this Board, since it is a puppet organization. Which is directly the opposite of what we were told would be the case.

    Again, back to the Trust issue. We have none now.

  • The problem is that we won’t know if it will work. When CentOS matched the RHEL point releases we knew that an RPM/driver targeted for RHEL
    8.2 has a good chance of working on CentOS 8.2 – but that versioning match is lost with Stream. So vendors will either have to produce another version of their RPM for CentOS 8 Stream (and continuously check to see if it needs to be updated) or, more likely, just not bother to support CentOS. It already happens – HPE won’t support CentOS, but they do support RHEL and those RHEL RPMs work with CentOS. The only Linux they support is RHEL, so we’re stuck with our HPE kit.

    So instead of keeping everything back for a point release, the packages are set free once they are ready. Stream is a rolling release. And that’s fine, but it’s not what people thought they were getting when committing to CentOS. It has always been promoted as point release compatible with RHEL and that was it’s main attraction to many people.

    A separate question. Will a point release of RHEL 8.x be directly a snapshot of 8Stream on a specific date? Or will RedHat pick and choose which versions from 8Stream they put into 8.x? i.e. Would it be possible to clone the 8Stream tree on the date that, say, 8.6 is released and call it 8.6.stream – would 8.6 be the same as 8.6.stream?

    P.

  • I don’t know if I’d call SIGs a playground, but they’re certainly an important venue for communities to explore variations.

    I think maybe some of the nervousness about CentOS Stream comes from RHEL
    seeming opacity on its development model. As one of the architects of our development process I’d be happy to field questions. I’ll start by making a two points in case they’re in any way unclear:

    1. Everything that goes into RHEL lands upstream first, then the patches are backported into the RHEL releases.
    2. Most of the work we do or plan on doing is in bugzilla.redhat.com. If you go into the RHEL8 product queue today and file a bug you’ll see “CentOS
    Stream” as a “Version” where an issue is encountered.

    I think what a lot of people are concerned about is the rolling-release

    Generally if they follow the ABI guidelines I would expect it to work. Those are here: https://access.redhat.com/articles/rhel8-abi-compatibility

    For loadable kernel modules there’s a kernel ABI.

    It’s regrettable when an ISV gets fixated on minor release versions and won’t recognize forward compatibility. This is generally more of a matter of policy rooted in legacy than a technical limitation.

  • Cool, I understand where you’re coming from. If the world remained static after this announcement I would be more concerned about this scenario. As it is, we’re in a dynamic space, and CentOS Stream will be a place that hardware vendors can participate as well.

    It’s certainly a change.

    RHEL is developed according to a schedule that keeps us delivering minor releases on a 6 month cadence. That includes a period of time when we’re putting the finishing touches on one release and simultaneously working on the next. Folks on the CentOS Stream team can speak with more authority on what the intended alignment points are (I know what makes sense from my perspective, but there may be other considerations I’m unaware of).

  • Hey,

    In my view, I think CentOS Linux 8 should be kept along with CentOS Stream
    8.

    Their purpose is different and I can’t see why they cannot coexist.

    There are production environments where even CentOS Linux 7 repositories are intentionally delayed with an extra month for non-security minor changes, just to be sure things are as stable as possible. I cannot see that moving from CentOS Linux 8 to CentOS Stream 8.

    But there are of course cases where users would want to move.

    If the two distributions would coexist, over the years, you will understand the need for users moving or not and act accordingly, in the interest of the community.

    So far, it looks like it will just backfire even for cases that could actually be quite prone to moving.

    Regards, Alex

  • +++ Rainer Duffner [2020-12-08 15:42:20]:

    That applies to IBM/RHEL too. They aren’t paying for every bit of software that is packaged in the distribution.

    So they can’t use that argument against others.

    Also people who are currently not paying are not going to suddenly start paying for RHEL subscriptions.

    A non-paying CentOS user is not a real loss for RHEL. But people dumping CentOS for a non-RHEL clone is definitely going to impact their future revenues as they are losing mindshare/goodwill/easy migration etc.

    Regards,

    Kingsly

  • It might or it might not. But you can’t say to people in good faith anymore that it will be as stable as the current RHEL release.

    I have recommended CentOS to my customers as way to get going and also recommended getting the subscription for RHEL when possible afterwards.

    After the change I lost my last argument for not going to Ubuntu LTS
    instead. Lot of companies I deal with have already done that. Problem is that with Ubuntu being in developers/users workstations that is what people mainly want without good arguments for something different.

    This might serve the way to reduce amount of RHEL subscriptions in future. If this will happen, that I don’t know. What I do know is that I
    don’t have any arguments left for getting people started on RHEL/CentOS
    route.

    -vpk

  • I agree with you 100%. I was almost laughed at by vast majority of Linux professionals when I say I use CentOS exclusively, because 90% of them uses Debian or Ubuntu, and around 99% of developers does the same.

    I see this developing like this:
    All/most of the hosting companies will stop offering CentOS in favor of Debian-based distro’s (I doubt they will try other RHEL clones) and the number of people who *HAVE* to learn CentOS/RHEL will dwindle, leaving only those who work for RHEL customers, reducing Red Hats subscription pool.

    I managed to filter all of my feelings and thoughts in two sentences:

    Majority of CentOS users only care about “99% binary compatibility with
    *upstream* distro”. Take that away and entire Red Hat opensource model and support is gone, same as Oracle has very little following in Linux world.

    —
    Ljubomir Ljubojevic
    (Love is in the Air)
    PL Computers Serbia, Europe

    StarOS, Mikrotik and CentOS/RHEL/Linux consultant

  • Il 08/12/20 22:12, Nicolas Kovacs ha scritto:

    I’m migrating away from CentOS since 8 released. The possibility of CentOS drop was enabled since IBM acquired RH. Johnny says that this is not an IBM decision, that a RH decision? Then it’s worst.

    My new ship (that was my old ship) will be debian and Ubuntu.

    Thank you Johnny for the hard work.

  • What will be the incentive for vendors to participate? Sure you can talk the corporate talk about opportunities and ecosystems, but the bottom line is that it requires investment (at least in time) when they could just continue supporting RHEL point releases, or possibly every other point release.

    I understand that the reason HPE, for example, don’t support CentOS is that there is no verification suite to ensure compatibility. Since CentOS is a different beast to RHEL now, are things like that going to looked at?

    Yes, yes it is. It’s a major change in philosophy for the distro. It’s a change that should not have been thrown at the community in such a way. There are ways of delivering and transitioning; there are such things as change managers to bring the community along with you. Working to make this change for CentOS 9 in 2023/4 could have been delivered without much backlash – I don’t think I have seen a single positive comment about this other than from people directly involved with RedHat/CentOS. There’s politics and corporate managers behind this somewhere.

    P.

  • Absolutely true. They are already ignoring what a real high quality distro is, and pushing into production things that are clearly unsuitable for the job.

    Regards.

  • And worse: Without CentOS being used widely (junior) admins will have even less experience dealing with RHEL. This will result in lower quality deployments of RHEL, reducing overall security of those systems and it will reduce RHEL adoption, hitting RH/IBM where it hurts them most.

    I am seeing this in practice already with juniors – they all use Ubuntu on their personal systems and they hate having to deal with RHEL. And their opinions matter in the long run.

    peter

  • And exactly the same applies to senior (or retired) admins on their home computers. My main home machine runs about a dozen testbed VMs, DHCP/DNS for the home network, Amanda, NFS and Samba for other machines, ownCloud, Apache, Zotero and DokuWiki for the family. I want a stable server under that lot, not a beta release.

  • Yes, and many things work well. My most recent issue was that kit supplied by HPE (sorry, it’s pain is stuck in my mind) had a RAID
    controller that needs a driver disk during install – doing the install time drivers is not a problem, the problem is that they don’t support CentOS, hence I had to use a RHEL driver and out of the 5 available for RHEL7/8, only one of them worked with a CentOS release. HPE support don’t want to know because they don’t support CentOS.

    I know this comes under the heading of “Corporate RedHat Policy”, but is RedHat going to do the right thing by CentOS 8 Stream to the level of lobbying other behemoth corporations such as HPE or Dell to support it?

    P.

  • Am 08.12.20 um 15:06 schrieb Rich Bowen:

    JFTR: I don’t have a problem with this change per se. I just don’t like this “Roma locuta causa finita” attitude and the fact that this happens for CentOS 8 right in its lifetime.

  • IMHO, if you based the most critical part of your infrastructure on CentOS, you did it wrong.

    The CentOS project doesn’t owe you anything for any reason. It’s completely your responsibility to support CentOS in your working environment and you have no right to pretend, for the developers, to mantain any commitment whatsoever to the project.

    We use CentOS on almost all of our hosting machines at the company I work for, and I’m not that afraid of this announcement. We will start to test the stream branch soon, to see if it continue to fullfil our needs with a good enough stability and, maybe, take a decision during the next year if we want to continue with CentOS or not.

    We also just switched to CentOS 8 from CentOS 6 spending around 6 months of work in doing so, but the most important part of our infrastructure is on paid RHEL
    licenses (i.e. hypervisors).

    CentOS was _always_ a best effort project done on a volunteer basis, if it was stable enough to build a company on top of it, good! But always remember that, in the end, you are getting what you paid for.

    Andrea

  • Gotcha!

    And Red Hat takes the effort on itself to mantain what it’s published on RHEL for you to use, indirectly, through CentOS; including fixing bugs, reporting things upstream and being (directly or indirectly) involved in projects they package in the various RHEL
    releases.

    How many of us, complaining here about a supposed “breach of trust”, are involved in making CentOS better and not just taking what others do and make money on top of it? I never participated in anything CentOS related, I happily use it but you should know what you are buying when you choose to use a (once) volunteer-based project.

    Andrea

  • It’s your own problem. I have opened at least 10 bugs on CentOS’s bugzilla and at least 30-40 (not counting the docu )bugs on RH’s bugzilla providing necessary details for identifying bugs and misconfiguration.

    Best Regards, Strahil Nikolov

  • I do not see you or anyone else as arrogant. Any harsh words, especially from Johnny can be attributed to frustration of not getting his view across, and I do not fault him for that.

    But harsh fact remains that CentOS Linux, a free clone of RHEL, is being slaughtered, and as far as I am concerned that “Stream” thingy can be renamed to RHEL Stream because it has no resemblance to CentOS Linux I
    know and love.

    CentOS Community wants free clone of stable and respected RHEL. That is only thing that we can use to compete with Debian/Ubuntu which have vanila kernel (it takes long text to explain to people what RHEL kernel is or is not), proprietary/non-GPL drivers, codecs and apps Red Hat does not want to add to RHEL, EPEL or Fedora. Majority of Linux users does not want to bother with 3rd paraty repositories, convoluted ways to add some things (that sometimes clash with base packages), etc. That is why CentOS or even Fedora was never accepted as Desktop/Laptop, and I am loosing desire to fight for CentOS’s place under the sun.

    I was actually planing to convert several C6 and C7 servers to C8, and now I am happy I did not waste my time on it. I even wanted to lobby to move servers of Mensa Serbia to CentOS (8), but those are now DEFINITELY going to be all Debian which my co-admin wishes, and it will be place where I will start learning Debian way in preparation for “Day of Sorrow” in 12 months.

    So with “CentOS Linux” gone, it is either some other clone (which will take a lot of fate I do not have at the moment) like Springdale or newly started Rocky Linux, or Debian Stable

  • Am 08.12.20 um 19:20 schrieb Alan Mead:

    These are exactly my thoughts of what will happen.

    This has already happened-
    Just take a look in Oracle’s yum repository and you see the available options.

  • Yes, we do. Let’s not forget that. Who knows? Amazon may see this as another opportunity and in 5
    years Amazon Linux will be the standard. Or Ubuntu, with its use of ZFS
    rather than playing again with btrfs–RH dropped that, but now it’s the default in Fedora, so it may make a reappearance.

    It had a good long run. Or maybe stream’s differences will be minimal, and almost nothing will change.

    Regardless, let’s not forget that we *do* owe everyone who worked on CentOS
    lots of thanks. (Though my current job is a FreeBSD shop, we have some things on CentOS that have packages for Linux but not FreeBSD.)

  • Am 08.12.20 um 22:30 schrieb Frank Cox:
    Oracle was/is much faster in releasing updates, point releases and releases. They don’t need CentOS to get OL going.

  • As CentOS Stream grows, I expect many companies who sell hardware will become active members of the community.

  • Probably, but this is not the point. The value of CentOS is that in essence it is identical to RHEL.

    This allows its use in multiple scenarios which the new CentOS Stream will not be able to support any more, due to its nature.

    You may want to read comments at:

    https://blog.CentOS.org/2020/12/future-is-CentOS-stream/

    and:

    https://www.change.org/p/CentOS-governing-board-do-not-destroy-CentOS-by-using-it-as-a-rhel-upstream

    to realize why this change will make it unsuitable in most of its current usage scenarios.

    Cheers, Nick

  • Gregory Kurtzer, founder od CAOS Linux that later changed name to CentOS
    is starting new RHEL clone: https://github.com/hpcng/rocky

    —
    Ljubomir Ljubojevic
    (Love is in the Air)
    PL Computers Serbia, Europe

    StarOS, Mikrotik and CentOS/RHEL/Linux consultant

  • Hi Brendan,

    This point is *critical*, so I apologise in advance for the lengthy post, *you* are breaking the kernel ABI between RHEL8 and Stream.

    One of the main unique selling points of RHEL is the stability of it’s kernel ABI. No other distro provides this. The very nature of rolling kernel updates in Stream breaks the kernel ABI and breaks compatibility between RHEL8 and Stream. What works on RHEL8 may not work on Stream. At the kernel level, the two products diverge in fundamental compatibility and are not compatible, are no longer the same.

    How bad is this divergence / breakage? Well, we know the kernel ABI will change from time to time, almost exclusively at new point releases due to the massive backporting work that goes into the RHEL kernel. And this is fine, we know it’s coming, we know when it’s coming, and we can plan for it’s impact. It’s a price well worth paying.

    To put this in context, at elrepo I currently help maintain around 50
    3rd party kernel driver packages for RHEL8. When RH released RHEL8.3, every single package in our repository broke due to changes in the kernel ABI in the 6 month period between RHEL8.2 and RHEL8.3. It’s not ideal, but we accept it as a price we pay for the otherwise excellent stability of the kernel ABI during the proceeding 6 months. As I said above, we know it’s coming, we know when it’s coming, and we can plan for it.

    Now contrast that with Stream. Every kernel update in Stream has the potential to break the kernel ABI causing packages built for RHEL to break. We don’t know when that may happen (only that it will), we don’t know how often it will happen, we have no idea which packages it will break. and we have no way to fix it. Consequently, elrepo has been unable to support Stream kernels.

    It is not just elrepo’s users that the Stream kernels will affect. All OEM hardware manufacturers releasing 3rd party driver rpms as part of Red Hat’s Driver Update Programme or otherwise will be similarly affected, and their driver updates will not be applicable to or compatible with CentOS Stream. In fact, RHEL’s own driver update packages will likely need rebuilding against each Stream kernel update, although presumably you are in the unique position of being able to integrate any changes directly into the Stream kernel negating the need for driver updates and to mitigating the impact to yourselves.

    Luckily the solution is simple – do not include rolling beta kernel updates as part of Stream – rather just ship regular RHEL8 kernel updates that retain kernel ABI compatibility rather than breaking compatibility of the Stream product. I see no compelling reason to ship rolling kernel updates to Stream – it’s not like I will ever be able to commit a patch and have you accept/merge it, and it’s totally useless for me as a developer as I can’t develop anything against a constantly moving target, and anyone who does require access has always had that available through their Red Hat OEM Partner/Developer accounts. Shipping rolling kernel updates in Stream adds no value and simply breaks compatibility between RHEL8 and Stream at the most fundamental level. If you are able to retain kernel ABI compatibility between RHEL8 and Stream kernels, then we (and other OEMs) will be able to continue to support Stream users, otherwise Stream users will have to look to alternative solutions.

    Phil

  • Maybe offering 2 kernels in stream may solve your problem? A “latest point release” and a “rolling version”? I realize that this may cause issues with packages that really need the new kernel features….

  • Oh, that’s right, it’s 2020, the dumpster fire of a year.

    >
    Retired sr. Linux admin here also, also running CentOS at home. Moved to
    7 this past summer (really dislike sstemd, hung onto 6 as long as possible).

    What I find outright offensive is that I see someone posted the specs for the Board… and #2 was “community outreach”.

    Can someone point me to a post, ONE SINGLE POST, before this announcement, saying that this was being considered? That this might possibly happen?

    Show me that this was not just presented to the community as a fait accompli, non-negotiable.

    mark

  • Perhaps using the Red Hat compatible kernel (if that’s what it’s still called, I haven’t followed any recent naming changes) from Oracle Linux could be considered? Not that this isn’t messy, but might provide what’s missing?

  • OK. We get it. We all get it loud and clear. You’re pissed off.

    There’s two things you can do about that –
    1) accept reality and start making plans to deal with it, or
    2) continue to whine and lash out at people who are probably feeling worse about the situation than you are, in which case I have to question whether you actually have the maturity to be able to administer a single machine, let alone any kind of IT facility.

    Cheers! (Relax, have a homebrew)

  • I think it is nonetheless not needed to start insulting people.

    We should try to keep the discussion friendly and technical.

  • Thanks Mark. We were told that the RedHat folks that matter (maybe) are monitoring this list, so I’m sorry if other folks here don’t want to hear it, but I’m not going to stop. I feel I am speaking for many, many others also.

    We *are* a community, and we should be heard.

  • Hi Phil,

    Thank you for these details, I really appreciate you taking the time to share this. It sounds like for your use-cases interim kernel updates are somewhere between not-useful and harmful, but I know for others it’s actually a benefit. As a temporary workaround, elsewhere in this thread somebody mentioned they’re already cherry-picking which updates to use and which to skip. The only trouble is when a kernel update comes out with both things you want and things you don’t want. So a better approach is called for. While I’m not sure how we’ll get there, it seems like the mutually satisfying end result would be one where third party binary drivers work with CentOS Stream kernels. Let’s see what we can do.

  • Hi Brendan,

    I support what Phil said. Guaranteed kernel ABI compability between CentOS Stream and current RHEL minor release would resolve most of my concerns (i.e. supporting 3rd party driver rpms).

    However this would disallow updating the CentOS Stream kernel as it is currently done. Maybe offering both, i.e. allow to optionally stay on the current minor release kernel version (including its updates) is feasible?

  • Hi Brendan,

    Thank you for your response, I am feeling more encouraged.

    As Peter eludes above, there are a number of ways this can be achieved. A separate repo/channel could be used to provide Stream kernels as an opt in update to the regular RHEL8 kernels. The main technical issue to overcome will be the fact the Stream kernels are of a higher NVR and the RHEL8 kernels, so it is difficult to see how the Stream kernel can be the default as they are now.

    Anyway, it’s not my job to solve these technical issues for you – you have plenty folks far more capable than I, that I’m sure can figure this stuff out given a commitment and clear direction from above.

  • Define stable. For most practical definitions of that and for most use cases, it absolutely will be. For some it won’t be, but I think there will be few actual cases where it won’t be _and_ CentOS Linux rather than RHEL
    _was_ acceptable.

    I don’t see why that would change. Or you may be able to get them started on RHEL in some new cases.

  • Oh, this one is easy. Because *this is how Red Hat is telling vendors to support RHEL point releases from now on*. I know it’s easy to get lost in everything else, but this is a huge pivot in RHEL development focusing more directly on CentOS.

    (Although as I understand it there will also be cases where code supporting new hardware is embargoed until a release date, which complicates things in some cases. That doesn’t change the overall new picture though.)

  • CentOS Stream will not be a “beta release”. That’s not how RHEL minor release development works. I personally think that it’s going to be stellar for your exact use case.

  • Understanding the flow of packages, is it a fair comparison to say that moving forward:
    Fedora packages could be considered alpha/beta releases of apps CentOS/Stream could be considered beta / Pre-release / Release candidates of packages / partially stable RHEL official releases would be considered final release / stable

    Where as before (done 12/2021)
    Fedora Packages would be beta / pre-release then RHEL and CentOS were final release / stable  – one with commercial support and the other with community only support.

    Is that accurate?

    —
    Christopher Wensink IS Administrator Five Star Plastics, Inc
    1339 Continental Drive Eau Claire, WI 54701
    Office: 715-831-1682
    Mobile: 715-563-3112
    Fax: 715-831-6075
    cwensink@five-star-plastics.com http://www.five-star-plastics.com

  • I doubt very seriously that changes made for point releases are ever considered beta quality ..

    Do the packages added betweren RHEL 7.8 to 7.9 or from RHEL 8.2 to RHEL
    8.3 differ so much? Not really. Look at the differences in the packages. For the most part .. except for some desktop rebases and the kernels .. the ABI/API stay the same.

    There are some rebases of some packages, but not very many.

    I keep trying to say .. no one is rolling in packages here straight from a new Fedora version or from Rawhide. These changes are point release type changes. That is the type of changes you see from 8.2 to 8.3 or
    7.8 to 7.9 … not major changes.

  • I would say that everything is much higher quality than that. We release whole non-beta releases of just Fedora.

    But particularly, no, CentOS Stream isn’t beta. Packages landing in Stream have already passed QA and gating.

  • Ah yes, that’s a great call-out. I’m not sure what the plan is there (or if there is one), but to me it seems like the sort of thing a SIG would build.

  • Well, yes, about 10 years too late for those discussions I’m afraid ;-)

    And besides, why on earth would Red Hat remove support for older hardware that you (understandably) no longer want to commit resources to maintaining, only to turn round and commit resources to maintaining them in a SIG? That’s why you guys reached out to us (elrepo) in the first place.

  • Il 2020-12-10 04:55 Brendan Conoboy ha scritto:

    Brendan, can you clarify the following points?
    – are you going to keep stable ABI between Stream kernel releases, or should we expect each kernel to break 3rd party drivers/modules?
    – what/how many synchronization points are going to be with RHEL
    releases?
    – what about security updates? Will they be released *before* the corresponding RHEL secure patch, or should we expect the (slow) current update cadency?
    – is an upgrade path from Stream-8 to Stream-9 planned, or the usual
    “total server rebuild” will be necessary?

    Full disclosure: the main CentOS point was to be 100% compatible, down to the specific kernel used, with RHEL. To get that, we lived with: a)
    comparatively few packages, b) not-working yum security-only updates and c) very restrictive selinux policies.

    I am heavily invested in CentOS/RHEL ecosystem and I opened many bug reports/enhancement requests in the past years, so I would really like to continue using CentOS. However, using Stream seems to removing the key selling point (ie: total RHEL compatibility) without clear benefit.

    Thanks.

  • Yes, but the calculus has changed a bit from 10 years ago, too, no?

    Note we’ve moved from me discussing facts about how RHEL works to my unauthoritative opinions on matters ;-) My point is really only this: if past decisions no longer make the most sense in the context of new events, we need to redesign. To the extent we can make things better we should make them better. For this topic it’s probably too early to tell if or how things should change in light of the upcoming changes, but it’s clear that handling it will make CentOS Stream adoption an option for more people who use CentOS today.

  • I’ll take a stab at it, though I’ll note Josh Boyer has already provided a few answers to similar questions…

    All our kernel changes are implemented against the kernel ABI- there is no point in time during release development when we have interim changes in the kernel that ignore the symbols in the whitelist. So basically if your experience of going from one minor release to another has been smooth, the incremental kernels between those two releases would also tend to run smooth, assuming whatever motions happen with the 3rd party drivers/modules behind the scenes continue to happen (for example, rebuilding from source).

    I’m not sure I’m interpreting your question correctly, could you restate?
    I don’t want to hit you with detailed process information only to find out I’m answering the wrong question!

    RHEL development prioritizes CVE resolution in support streams, followed by current release update streams.

    Upgrades are important. We continue to invest in the major release upgrade tooling introduced with the launch of RHEL 8.

    Full disclosure: the main CentOS point was to be 100% compatible, down

    Thanks for the bug reports :-) I hear you on RHEL compatibility. With my OS developer hat on I think CentOS Stream will be more RHEL compatible, but if I put on my old dusty ops hat I can understand why it might not look that way right now.

  • Me too installing CentOS 8 as hypervisor on Dell M610 with oVirt 4.4.3 (not possible to use it as ovirt-node-ng due to the missing kernel mode and impossibility to inject DUD in oVirt NG Node): now upgrading to 8.3 using kmod-megaraid_sas-07.714.04.00-1.el8_3.elrepo.x86_64
    Thanks!

    Gianluca

  • Il 08/12/20 18:19, Marc Balmer via CentOS ha scritto:
    No and this is not started with the current discussion.

  • El mar, 8 dic 2020 a las 13:44, Nicolas Kovacs ()
    escribió:

    +1 But for some It centennials progress is about destroying the present. Smoke as a Service.
    —
    —
    Sergio Belkin LPIC-2 Certified – http://www.lpi.org


  • I expect them to leave. In the real world, we had EXTREMELY limited windows to update servers and workstations. To expect people to do daily updates is asking for management, as well as the users, to scream bloody murder. Most are *not* that technical, and will start blaming the update for something else not working, and management will hear *them*.

    I was chased off RH after RH 9, when it went to pay for licenses (and I
    was “between positions”), and came back, because I *like* the RH
    architecture. but I’m considering ubuntu now.

    mark

  • I see, so your response to WHY DIDN’T YOU POST HERE, AND EVEN LET
    EVERYONE KNOW THIS WAS BEING CONSIDERED?! is “sorry you’re pissed, tough”… no apology for NOT WARNING US, no nothing.

    “Community”? Obviously you left us behind.

    mark

  • ROTFLMAO!

    Reminds me of when Anderson Consulting, now Accenture, was going start a spin-off in Europe called Monday Morning… but disgruntled ex-employees got the domain name before they did (they announced before getting the domain name), and the ex-employees had two fingers dancing on a desk, singing “we got your name, we got your name….”

    mark

  • I agree with you matt, the community should be listened to, it is not possible that it is the decision of a few for something, it is that CentOS is part of the OpenSource ecosystem.

    Regards,

  • Il 2020-12-10 18:40 Brendan Conoboy ha scritto:

    Hi Brendan, no, it was not intentional – I replied from the smartphone and I accidentally dropped the CentOS-devel list. I’ll reply quoting the entire conversation to let others read your useful information.

    Thank you for the clear replies. Regards.

  • Look at Rocky Linux rockylinux.org It is set to become what CentOS was before it was sucked in by RH and sold to IBM, a Community Enterprise OS.

  • Back in the Before Times and RHEL 7 was at .1 or .2 I had a persistent kernel oops on a set of RHEL7 hypervisors. Since we had fairly well tricked out RHEL support licenses I opened a ticket and within a couple of weeks I had confirmation that yep, there was a known issue, and there was a fix entering testing. I asked for access to the fixed kernel. I was told no. I asked if there was a particular kernel version I could deploy temporarily until the fix was released. Silence. I asked for a bug ID so I could maybe use that to figure out what kernel I could deploy until RH released the fix. Silence. Meanwhile hypervisors are oopsing on me because the project didn’t want to deviate from the vendor baseline.

    Then I learned about CentOS Plus. I reprovisioned a machine, picked a Plus kernel, and happy sailing. The project decided functionality was superior to arbitrary compliance in this case. Rebuilt the rest of that rack to CentOS 7 and never looked back. Or bothered renewing as many and that level of support because the one time I really could have used it it was effectively denied.

    Sometimes basing the most critical part of your infrastructure on CentOS was the only way forward.