27 thoughts on - Current RHEL Fragmentation Landscape

• I disagree with you. Both Rocky Linux and AlmaLinux are in a very comfortable position, and they will likely stay that way.

  my predict is that they will continue as a #rebuilder / #freeloader, writing software is a hard work.

  SuSe hardfork will probably be only an stable version of CentOS Stream.

  #offensive terms to the community :-), hide hat wrote it.

• We’ll see, Rocky Linux wants to stay a 100% rebuid of RHEL. Red Hat could be tempted to hide more of the sources of RHEL, the parts not under GPL.

  This won’t hurt AlmaLinux as much because there is no real need to be 100%
  RHEL compatible. It must be 100% compatible with itself, sure.

  We shouldn’t forget that Red Hat didn’t write all the software for RHEL. They wrote some parts of it but “freeloaded” 90% from the community. I’m also part of this community and never asked for a cent from Red Hat. The deal is quite clear, isn’t it?

  These terms are a shame for those who speak them out.

  Simon

• No, they didn’t.

  That term was bandied about on social media by people who were speculating about the reasoning behind discontinuing the practice of debranding and publishing packages from RHEL minor releases.

  Mike McGrath responded to the use of that term by social media personalities to explain that the only group that Red Hat (for better or worse) considers freeloaders are large businesses who keep a small number of licensed RHEL systems so that when they have problems in their production network (which isn’t running RHEL), they can reproduce the problem on RHEL and ask Red Hat for support.  That practice is dishonest and abusive.

  If you’re not doing that specific thing, then Red Hat is not calling you a freeloader.

• I subscribe (pay) for a lot of things personally. Music, Movies, Anti Virus, VPN, Storage, etc. But for my business, I do not want to pay Red Hat, Zimbra, or Google Workspace. Why ?
  Because the general rule seems to be Oh! You are an individual, we will offer you affordable/free service What! You are a business, we will offer you extremely ‘unaffordable’ service. Because being a ‘business’ by default means you have a ‘lot’ of money to waste.

  Just my two cents.

  —
  Lee

• Fwiw we pay for Google Workspace. I think there is a pricing issue though, but I guess changing it would affect their income from their bigger customers.

  We just have 5 servers, and don’t want any personal support. We’d be fine to pay what we’d consider a reasonable fee I think. I contacted Redhat to ask about their licensing and if we could fit somehow into it (i.e the personal support & 16 machine type license), but they could never give us a straight answer, so the implication was we couldn’t be certain we weren’t breaking any t&cs. I also thought maybe they’d make an offer or something, but never did. The costs are just too high for some people for what they are offering.

  The problem now, is what I think people are happy to pay for is trust, which is gone. If we’d have known in advance, we’d have made other choices.

  Ian

  CentOS mailing list CentOS@CentOS.org https://lists.CentOS.org/mailman/listinfo/CentOS

• Sorry for being too critical. I hope we have a better understanding between us (customer and provider).

  Thanks
  —
  Lee

• I have about the same number of CentOS 7 servers and have never felt the need to call RH for help. At most, I’d check online for a solution and find it’s maybe behind RH’s paywall. If anything, I work the problem on my own (maybe using a specific package’s own support list) and supply the fix to Bugzilla (as well as upstream). Now I’m feeling reluctant to do that.

• I’m not a Red Hat employee, so I’m not positive how they would respond to that.  But, speaking as a customer who has worked with numerous enterprise support agreements over several decades, I want to suggest that the issue isn’t that Red Hat assumes that businesses have a lot of money to spend, it’s that they’re targeting a set of the market that you might not be in right now.

  From my point of view, Red Hat doesn’t really sell software. They give away software.  All of their software is available at no charge, typically in an unbranded release.  What Red Hat sells is support.

  I don’t mean helpdesk style “support-me-when-something-breaks” support. 
  Support isn’t something that exists only during incidents, support is a relationship. It’s periodic meetings with your account manager and engineers. It’s discussing your roadmap and your pain points regularly, and getting direction from them. It’s the opportunity to tell Red Hat what your needs and priorities are, and helping them make decisions about where to allocate their engineers time to address the real needs of their customers. It’s setting the direction for the company that builds the system that sits underneath your technical operations. That kind of support is what makes RHEL a valuable offering.

  If you don’t need the kind of support that comes with enterprise offerings, then by all means, use the Free Software that Red Hat provides to the community.  But don’t make the mistake of thinking that Red Hat is trying to mlik businesses simply because they’re businesses. 
  Red Hat’s offerings are expensive because they’re enterprise-focused support plans.

• Does Red Hat give away software anymore?

  I am confused.  Last month Red Hat announced that the source code would not be published.

  an individual.  Companies do not pay tax on their expenses. That might partially explain the higher rates for commercial products and services.

  ————————————————————————————————————————————————————————————————-

  I am not an expert network manager.  I am a physician that used CentOS 8
  on my three practice servers until the big “rug pull.” At the time, I
  had a choice between switching to the Stream or Oracle Linux 8.  I went with Oracle Linux 8 and had no complaints.  Some have suggested that the evil Oracle will execute the same IBM rug pull.  I considered that. 
  That concern is a non-issue now.

  The spirit of GPL was meant to force sharing and prevent the commercialization of the volunteer work of many.  At the time, I was confused about why IBM purchased Red Hat for an astronomical amount. 
  Well, it is clear now.  As the readers know, there is a significant defect in the GPL: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model
  <https://sfconservancy.org/blog/2023/jun/23/rhel-gpl-analysis/> The terms of the license are enforceable, not the spirit

  I think the Rocky Linux workaround will eventually fail.  I expect IBM
  already has a plan for all contingencies.

  There is reason for anger.  Is there a reason for hope?

  frank saporito md

• +1 Frank

  Personally I am moving all my workloads to anything but. It’s clear the direction that Red hat is taking and so be it. I’ve seen it multiple times with open source projects that just seems like greed kicks in and it’s all about making the most $$ that they can. Oh and let’s be clear, for anyone saying that Red hat wasn’t making money is crazy talk, they have always made enough and that was clear when IBM bought them for 34B. Really for the most part they have become another Oracle.

  The main advantage for me with RHEL was the long support cycle, plenty of distros do 5 year but the 10year helped for a lot of things maybe someone will come along and do just that.

• ++++++++++++++1

  Frank Saporito nailed it, walks like a duck, quacks like duck, it’s a duck!

  There are dozens of alternatives and better community projects.

  IBM/RedHat will learn the hard way, as subscriptions decline, and the user base decreases, and CentOS/Fedora communities will end up suffering the most because of greed.

  —–Original Message—–
  From: frank saporito
  Reply-To: CentOS mailing list
  To: CentOS@CentOS.org Subject: Re: [CentOS] Current RHEL fragmentation landscape Date: Sat, 22 Jul 2023 11:55:28 -0500

  Does Red Hat give away software anymore?

  I am confused.  Last month Red Hat announced that the source code would not be published.

  Businesses can purchase in a tax-advantageous manner that you can not as an individual.  Companies do not pay tax on their expenses. That might partially explain the higher rates for commercial products and services.

  ———————————————————————–
  ———————————————————————–
  —————————————————

  I am not an expert network manager.  I am a physician that used CentOS
  8
  on my three practice servers until the big “rug pull.” At the time, I
  had a choice between switching to the Stream or Oracle Linux 8.  I went with Oracle Linux 8 and had no complaints.  Some have suggested that the evil Oracle will execute the same IBM rug pull.  I considered that. 
  That concern is a non-issue now.

  The spirit of GPL was meant to force sharing and prevent the commercialization of the volunteer work of many.  At the time, I was confused about why IBM purchased Red Hat for an astronomical amount. 
  Well, it is clear now.  As the readers know, there is a significant defect in the GPL: A Comprehensive Analysis of the GPL Issues With the Red Hat Enterprise Linux (RHEL) Business Model
  <https://sfconservancy.org/blog/2023/jun/23/rhel-gpl-analysis/> The terms of the license are enforceable, not the spirit

  I think the Rocky Linux workaround will eventually fail.  I expect IBM
  already has a plan for all contingencies.

  There is reason for anger.  Is there a reason for hope?

  frank saporito md

• Yes?  I’m not aware of any Red Hat software that isn’t Free Software.

  That’s not what they announced.  The major-release branch of RHEL’s source code is still published to the CentOS Stream git repos.

  I think it’s important to point out that Red Hat never published *all*
  of RHEL’s package source code.  For the first six months of any release of RHEL, they would publish de-branded source by essentially taking one artifact from each build (the src.rpm), unpacking that in a git repository, removing the primary source code archive, debranding what was left, committing all of that, and then pushing the result.  It was basically git as a fancy FTP.

  They’ve stopped doing that, in favor of publishing the major-release branch of the git repos for the entire primary support lifecycle of the major release.

  It definitely wasn’t.  GPL software can’t be made closed-source. Customers have to receive the source code (or an offer for it), and they have the rights that the license guarantees.  But GPL software can definitely be commercialized.

• source and their intent. RHEL is for all intents and purposes trying to restrict the source code with EULA’s/Licensing restrictions, you still have access to the source for paying customers but if you use it for a purpose that they disagree with *cough* rebuild, then they can terminate your account. I can list article after article clearly stating that is what they are doing –

  https://www.itpro.com/software/open-source/what-red-hats-source-code-restrictions-mean-for-businesses https://www.theregister.com/2023/06/23/red_hat_CentOS_move/
  on and on…

  again its a d1ck move IMHO, clearly you do not see the it that way but let there be no mistake what Red Hats intentions are at all….they no longer want anyone to be rebuilding their software and distributing it, ya know they need the money, lol….

• I’m not dancing around anything.  I’m discussing the objective, verifiable facts of what they did, some of my opinions on that, and not Red Hat’s intent, because that is a thing that we cannot know. 
  Restraining commentary to the things that we can know might look like
  “dancing around” the subject to someone who’s accustomed to discussing the intent of others, but as an engineer, I tend to think that there’s nothing *wrong* with focusing on the facts and evidence.

• this is ok, but the worse thing is:  students and teachers get affordable/free service

  and other citizens had to pay unrealistic sums of money …

  (a) talking about money to waste is nonsense
  (b) think of the fact that this way residents get something affordable, which is absolutely fair;
  e.g. residents get 200 Mbit down/20 Mbit up unlimited for 30 dollars a month,
  ‘business’ has to pay for the same more than 100 dollars a month;

• I would like clarification on your recent post. There may be some nuances in your language that I need help understanding.

  Let me know if you disagree with any of these statements:

  1. Red Hat is no longer posting source code to git.CentOS.org. (ref: Red Hat’s June 21, 2023 announcement, https://www.redhat.com/en/blog/furthering-evolution-CentOS-stream, and Hackaday article published June 30, 2023, https://hackaday.com/2023/06/23/et-tu-red-hat/)

  2. Red Hat will release source code to partners and customers via the Red Hat Customer Portal. (ref: Red Hat announcement)

  3. Per Red Hat EULA, customers can not freely distribute the source code. (ref: Red Hat EULA)

  4. Red Hat’s policy decision has made it difficult (maybe impossible)
  for “clone” distributions to continue existing. (ref: Google “red hat source code”)

  5. Red Hat’s policy change contradicts the GPL’s spirit.

  The first four statements are facts (at least, I think they are.)  The fifth question is opinion.

  I am not engaging in a heated debate – just trying to gain understanding.

  I appreciate your consideration. frank

• Correct.  Red Hat used to publish a de-branded subset of RHEL source code there, and they’ve discontinued that process.  The current code for RHEL is now published to the CentOS Stream repos.

  Also correct.  This is the only channel through which Red Hat ever posted complete code for RHEL.  It hasn’t been changed.

  It’s a little more complex than that, but probably close enough for now.

  This is the point at which I think we start to wade out into the territory of myth.  It has never been possible to create a clone of RHEL
  from the code that Red Hat published.  First, because Red Hat doesn’t publish the information that would be required to create reproducible builds.  But more importantly, because RHEL has one life cycle per minor release, and distributions built from the old git.CentOS.org repositories had *at best* one life cycle per major release.

  CentOS Stream also has one life cycle per major release, and conforms to the interface compatibility guide for the matching RHEL major release.

  Distributions derived from CentOS Stream can have either lifecycles per minor release *or* one lifecycle per major release.  Unlike the old source publication process, they can have continuous or overlapping life cycles.

  Yes, this involves more steps than the old process.  The next natural question is whether the additional work is justified by the improvement in the outcome.  And from my point of view, that is a very easy “yes”.

  I understand that it’s confusing, but CentOS was never a substitute for RHEL, and never provided the benefits of RHEL’s model.  It is not the
  “free RHEL” that many users tend to think it was:

  https://fosstodon.org/@gordonmessmer/110648143030974242

  https://www.youtube.com/watch?v=tf_EkU3x2G0

  … and conversely, CentOS Stream is a much better stable LTS for self-supported systems than you might believe:

  https://medium.com/@gordon.messmer/in-favor-of-CentOS-stream-e5a8a43bdcf8

  As you acknowledge, that’s a subjective question.  I would say “no.”

  I think the entire history of the free-as-in-speech vs free-as-in-beer clarification is proof that we wanted to ensure the right to improve software if you didn’t like its limitations, not the right to give away software if you didn’t like its price.

  But I also think it’s important to acknowledge that the thing that rebuilders are asking for (the RPM source repositories) aren’t GPL
  licensed, they’re MIT licensed, which makes the question something of a non-sequitur.

• I think you’re simplifying too much here and I’m quite sure it’s not true that “RPM source repositories aren’t GPL licensed, they’re MIT licensed”.

  RPM source repositories consist of SPEC files, patches, documentation, helper scripts and more and for sure not all of them are MIT licensed. Many of those files are actually GPL licensed and as such enjoy the freedom that the GPL asks for.

  We don’t have to study every letter of the FLOSS licenses to understand what Red Hat is doing. Things are much simpler:

  Let’s assume Red Hat has written 15% of the sources of RHEL themself. That would mean they’ve got 85% of the rest of the source code for free from the worldwide community. Now, large parts of this community expect to also get access to Red Hats 15% of source code without any restrictions, the same way Red Hat got access to the 85% of source code from the community.

  Everything else is considered unfriendly, shameless and evil, by a large part of the worldwide open source community.

  Regards, Simon

• this is ok, but the worse thing is:  students and teachers get affordable/free service

  and other citizens had to pay unrealistic sums of money …

  (a) talking about money to waste is nonsense
  (b) think of the fact that this way residents get something affordable, which is absolutely fair;
  e.g. residents get 200 Mbit down/20 Mbit up unlimited for 30 dollars a month,
  ‘business’ has to pay for the same more than 100 dollars a month;

• I was trying to stay out of this thread, but the reply was complete and utter nonsense…

  Nonsense. For years Red Hat freely published the complete RHEL SRPMs to their public ftp server. It *has* changed, a number of times over the years as Red Hat increasingly seek to make it harder for others to exercise their GPL rights.

  It’s not complex at all. The GPL absolutely allows recipients to freely redistribute the RHEL sources. Red Hat seek to prevent their customers from exercising their rights under the GPL by imposing agreements that allow them to terminate their contract and/or take legal action if their customers choose to exercise their GPL rights.

  Lets be very clear what Red Hat are doing (and to the best of my knowledge have always done).

  Of course it has. CentOS did it for years, and so did Scientific Linux and others. The GPL *requires* Red Hat to publish the full sources including “the scripts used to control compilation and installation of the executable.”

  If it is not possible to create a clone, then Red Hat are not in compliance with the GPL.

  Yes they do – it’s all in the SRPMs. I will concede that these need to be built in the correct order and thus within the correct buildroot (the skill that CentOS and others brought), but the information required “to control compilation and installation of the executable” is all there.

  Seriously? You are the only person here who thinks that.

  The GPL grants the rights to redistribute sources. It goes further, it specifically states:

  ” To protect your rights, we need to make restrictions that forbid anyone to deny you these rights or to ask you to surrender the rights. These restrictions translate to certain responsibilities for you if you distribute copies of the software, or if you modify it.

  For example, if you distribute copies of such a program, whether gratis or for a fee, you must give the recipients all the rights that you have. You must make sure that they, too, receive or can get the source code. And you must show them these terms so they know their rights.”

  Red Hat’s terms specifically contradict that and specifically prohibit:

  “using Subscription Services in connection with any redistribution of Software”

  in section 1.2 (g) (d) of their terms (below), considering it a
  “material breach of the Agreement” (IANAL, but that sounds like legal speak):

  https://www.redhat.com/licenses/Appendix-1-Global-English-20230620.pdf

  Now you may consider the above subjective, but in doing so you lose all credibility on this list.

  I understand the above has yet to be tested in a court of law, but there is no doubt it does not comply with any reasonable interpretation of the intent of the GPL.

  Further, there can be no doubt that Red Hat fully understands this too. Red Hat’s own definition of Open Source can be found on their website:

  https://www.redhat.com/en/topics/open-source/what-is-open-source

  “Open source is a term that originally referred to open source software
  (OSS). Open source software is code that is designed to be publicly accessible—anyone can see, modify, and distribute the code as they see fit.”

  yet Red Hat’s own terms explicitly seek to prevent this. And you. Gordon Messmer, think this is not contradictory?

• No, they didn’t.  Take a look at the planning guide diagrams, here:
  https://access.redhat.com/support/policy/updates/errata

  A RHEL major release isn’t a single lifecycle.  It’s a sequence of minor releases, many of which have 4 year lifecycles of their own.  Red Hat never published the updates for those lifecycles after the first 6
  months, which is why CentOS had long gaps with no updates every 6
  months, while they built a new release.

  If Red Hat had published *all* of their source, then CentOS could have continued publishing security errata right up until they were ready with a new minor release.  But that’s not the way that it worked.

  Few people, if any, were ever really concerned about the fact that Red Hat didn’t publicly publish the source for their extended support life cycles.  But, that’s what minor releases fundamentally *are*, so it’s weird to see users unhappy with arrangement today, in which the current sources for RHEL are published to the CentOS Stream git repositories, and the minor releases are treated the same way that extended support life cycles always have been.

  That’s not what I’m saying.  The GPL does allow RHEL customers to redistribute source code for GPL-licensed software.  The complexity is that the license doesn’t require Red Hat to continue business relationships and support users who choose to do that. And that most of RHEL isn’t GPL-licensed.

  No, it wasn’t.  CentOS’s maintainers were pretty clear about this when they were asked: https://www.spinics.net/lists/CentOS-devel/msg19564.html

  “CentOS was NEVER bug-for-bug compatible. … Sometimes CentOS shipped packages which did not have a particular bug because we could not exactly duplicate the build environment and other times we added new bugs because our build environment is not exactly the same… At best, CentOS has been “good-enough” compatible for a set of years ”

  And that’s only for the packages that they actually shipped. CentOS
  never reproduced the extended life cycles that RHEL provided, which are what make minor releases actually valuable. Without the overlapping life cycles, there’s just not a really good reason to have minor releases.

  To customers, yes.

  Red Hat goes above and beyond that requirement, by publishing the current state of the source code to the public, for both GPL and non-GPL
  software.

  See https://reproducible-builds.org/ for more information.  It isn’t all in the SRPMs.  As above, CentOS’s maintainers were always clear that it was never possible to reproducibly build RHEL.

  I don’t think I am, but it doesn’t really matter.  Most of RHEL isn’t GPL anyway.  Even if the subscription agreement only covered content that wasn’t GPL, you still can’t build RHEL from the GPL sources alone.

  The GPL isn’t anti-commerce.  It doesn’t prevent the sale of software. 
  The spirit of the license is that users should be able to modify the code and build new systems, which they certainly can do with RHEL source code.  There’s no secret sauce in RHEL that isn’t publicly available to users who want to build something new.

• After reading an unrelated thread, I want to make an additional comment:

  There are several reasons that I think that Red Hat’s subscriber agreement is in the spirit of the GPL.  One of them is that nothing is developed in RHEL minor releases that isn’t available to the public.  A
  RHEL minor release is just a snapshot of the major release branch (which is now used to build Stream), that Red Hat engineers continue to support.  Updates to the RHEL minor release are either a direct merge of changes from Stream, or backporting a fix if Stream has rebased a package which needs that fix.

  If Red Hat were doing development in RHEL minor releases that wasn’t published elsewhere, I would probably have a different view of thing, but they aren’t.  There’s nothing there that isn’t published elsewhere.

  The *software* is freely available to everyone, through Stream.

  RHEL is a *support* program that provides extended support to snapshots of the software (among other things), by applying patches that are generally available.

  Providing support is not a violation of the spirit of the GPL.

  Development all happens upstream, in publicly accessible repositories. 
  That’s a core part of my outlook on the subject.

• Once upon a time, Gordon Messmer said:

  This will not be the case for the second half of a RHEL major release life cycle, because the corresponding Stream will be EOL and no longer updated.

• As best I understand Red Hat’s “upstream first” policy: every patch applied to RHEL X.10 will either be a patch that they import from an upstream project, or (for patches that Red Hat develops) will be offered to the upstream project.  They’re not held in reserve for RHEL customers exclusively.

  So, they may not appear in the Stream git repo, but the patches are still publicly available through other channels.

  If anyone has examples of this not happening, then we can talk about whether the process is working as intended, and what that means.

• Am 26.07.23 um 00:52 schrieb Gordon Messmer:

  Honestly, you are mixing unrelated, or not relevant topics and arguments, and even misconceptions and forget to understand the problem at all. When done intentionally, its just a flashbang approach and this doesn’t contribute to clarify the actual new reality.

  —
  Leon

• I don’t see how that’s unrelated.  As I said earlier, on this point we’re discussing a matter of opinion on the “spirit” of the license, and mine is heavily influenced by the fact that RHEL is composed of publicly available software and patches.

• And neither is *not* providing support.