C7, Systemd, Say What?!

Home » CentOS » C7, Systemd, Say What?!
CentOS 42 Comments

I just updated a system – as in minutes ago, and log back in after it reboots, and this is in dmesg:
[ 88.202272] systemd-readahead[484]:
open(/var/tmp/dracut.fP4yj1/initramfs/usr/bin/loginctl) failed: Too many levels of symbolic links
[ 88.202515] systemd-readahead[484]:
open(/var/tmp/dracut.fP4yj1/initramfs/usr/lib/systemd/system/dracut-emergency.service)
failed: Too many levels of symbolic links

Anyone know what this is – some weird bug, a garbage message?

mark

42 thoughts on - C7, Systemd, Say What?!

  • I’m not sure why it’s trying to open anything in /var/tmp to be honest. Jacked up filesystem maybe? Granted I know very little about systemd except it sucks on levels that I can’t begin to explain.

  • systemd-readahead is just trying to pre-cache stuff into memory so boot time is faster. I’m not sure eaxctly what’s going on here but that message is typical of having a loop in your symbolic links (which can easily happen with relative links).

    I’m not quite sure what *exactly* is going on, but it looks like maybe dracut temp files didn’t get cleaned up properly and that they contain such a loop. I bet you can just rm -rf /var/tmp/dracut.fP4yj1.

  • Matthew Miller wrote:
    Thanks for the info. Now, why it shouldn’t have cleaned itself up when I
    gave it the reboot command… I see too many (that’s defined as more than zero) cases where systemd WANTS TO BOOT FAST, and doesn’t wait for things to finish – sush as not getting the hostname from dhcp, and so having to hardcode the name instead.

    Systemd, as I’ve said before, seems to be targeted towards laptops. Not servers. Not workstations. *bleah*

    mark

  • Mark stop with the flame baiting please.

    This is nothing systemd specific – and keep in mind /var/tmp is a persistent temp area unlike /tmp which as it’s tmpfs by default is of course emptie don boot.

  • I’m still thinking it’s a jacked up filesystem. I’m not sure what fs you’re using, though the default is xfs, but I’d look at dmesg and boot.log to see if the kernel is finding issues with the drives or just the fs. It’s also possible that server had been up a long time and RAM
    was funky. I’ve seen both of these happen before.

    As far as using systemd based systems on servers, a month or so back, I
    pushed a new C7 kickstart for servers we send to customers and haven’t seen anything to make me think systemd isn’t good for servers. That doesn’t mean it’s not a giant POS for administrators. If only they hadn’t jacked the syntax all to hell from initd, I might be slightly happier with it. That by itself has to be the most ridiculous thing any group of devs have ever done. And for no rational reason either.

  • I would wholeheartedly disagree. This IS something systemd specific. I
    have never seen init.d blow itself up over bloody symlinks. The readahead, while /possibly/ nice isn’t at all necessary on modern hardware. I want my hardware to boot consistently, not bomb like an Adam Sandler movie because of /symlinks/.

    But hey, call it flamebait if you want. I’d be willing to bet a year’s salary most admins hate systemd with a passion.

  • Now this is just silly. It didn’t “blow itself up”. Nothing blew up at all. There are just messages logged. There is no actual problem.

  • Mark Haney wrote:
    Yup. I have issues with just trying to find out the name of a service to restart…. But let’s not go on about it. I just had a throwaway, and really didn’t intend to start something like the last flame thread on systemd, that went on for weeks….

    mark

  • I’m sure James Hogarth meant that circular symlink chains are a problem for any program, not just for systemd.

    Now if you can show that systemd *generated* this chain, you might have a point.

    Since we have only one report of this, it seems unlikely that systemd is doing this. Surely we’d have thousands of reports of this is something systemd always did.

    Only the systemd-readahead process failed. It’s an optional component. systemd is clearly not “blown up” on Mark’s system, else he couldn’t have gotten to a login prompt.

    This optional component diagnosed an actual problem, that’s all.

    Explain then why a VM containing a given OS generally boots faster the second time on a given host than rebooting the same OS on the bare hardware running that VM host.

    RAM caches matter more today than they ever did, due to the vast disparity between storage access speeds in a modern system. Precharging the caches is still a good idea in 2017.

    Hyperbolic much?

    I think you’re basing that bet on data generated by an angry noisy minority.

  • With all my respect to Warren, I still decided to mention: noisy minority are the only ones everyone hears. There is big bunch of people who stopped saying anything as when one does, he gets in the middle of heated argument, gets shushed upon… And both sides of argument do realize that arguing will not change anything. Especially NOT on CentOS list. I am just mentioning that obviously visible minority has a bunch of invisible folks agreeing with them who decided to stop commenting on this, and some who fled elsewhere (and definitely didn’t change their minds). Sigh.

    Let’s close this theme, and go back to administering systems we love
    (whichever OS that would mean for you).

    Valeri

    ++++++++++++++++++++++++++++++++++++++++
    Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247
    ++++++++++++++++++++++++++++++++++++++++

  • In systemd fstab takes care of only rudimentary mounting. Most mounting is done through *.mount unit files. Type “mount” and you’ll see a bunch of other mounts that were implemented that way. Add your custom mounts by creating suitable files in /etc/systemd/system/*mount. (There’s also
    *.automount for creating demand-based mounts.)

  • Before this turns into another 200 email flame war about systemd .. this list is NOT the place to discuss if systemd is good or bad, nor whether it should be in a CentOS Linux distro or not.

    CentOS rebuilds source code for RHEL as released by Red Hat. If that source code uses GNOME 3.14 instead of GNOME 3.18 .. or if it uses mariadb instead of mysql .. or if it uses systemd or sysv init .. is not relevant to how we build CentOS Linux or what CentOS Linux is. If you want to influence what is in upstream RHEL (so therefore what gets released as source code, and therefore becomes part of CentOS Linux), Red Hat has mechanisms in place where that happens for both Fedora and RHEL. This is not one of those mechanisms.

    CentOS rebuilds the source code that is put out, nothing more. No one is making anyone use CentOS Linux, or like what it contains. If you don’t like CentOS, don’t use it. If you don’t like systemd but want to use CentOS Linux, use CentOS-6, which does not have systemd.

    If you want to create a CentOS-7 variant that does not use systemd, then start a Special Interest Group and create modified packages to use something else instead, much like the this group did with Debian:

    https://devuan.org/

    In the case of CentOS-7 .. you don’t need to create a whole new distro, you can just petition the CentOS Project Board to create a Special Interest Group to get access to CentOS Project controlled resources to build packages (and get them rolled into our mirrors, etc.) to use something other than systemd.

    But just whining about not liking content in CentOS Linux in general, or systemd in particular, is not productive. Use CentOS if you want, if you don’t that is fine. If you want something major changed .. this is open source and we provide mechanisms to make such changes (Special Interest Groups), so use them.

    I am NOT saying that Mark (or anyone else) is whining at this point .. I
    just picked the original mail in this thread to post this email reminding how CentOS Linux works and to suggest how something constructive might be done instead of another irrelevant (to CentOS
    Linux) ‘I like or I hate systemd’ thread.

    Thanks, Johnny Hughes

  • Kenneth Porter wrote:
    You. Have. To. Be. Joking. WHY? Why doesn’t systemd *look* at fstab and create what it needs on the fly? Why does it only “rudimentary mount”?
    What purpose does that serve? What goal is it trying to achieve by this?

    mark

  • It does that. Read the man page for ‘systemd-fstab-generator’, and
    ‘systemd.generator’.

    I think the biggest use I see is that your services can have dependencies on mountpoints.

  • Calm down Mark. You are overreacting. Systemd does generate mount units in the fly. Check the documentation:
    man systemd.mount tells you more. I would not call fstab rudimentary.
    /Louis

  • Perhaps I phrased that poorly. The idea is that fstab provides a minimal set of mounts to get off the ground. (My understanding, not saying that’s how it’s designed or intended.)

    This follows the packaging pattern that every unique setting goes in its own file. You don’t disturb or risk breaking other settings when you add a new setting, and you can separately package these settings without meddling with central files.

  • Kenneth Porter wrote:

    Ok, sorta. Thanks for expanding, all.

    mark “none o’ youse guys had to spend something like six hours
    Monday and Tuesday swapping 42 drives in a RAID from 2TB
    to 4TB, 6 unscrews, 4 screws….zzzzzzzz”

  • but will you contribute to building the non-systemd packages, and working out how to retrofit old sysV init back into everything via patches, etc ? every RPM that interacts with systemd will need to be
    ‘fixed’ to do it the old way, with init.d scripts. repositories like postgres, EPEL, etc won’t work, either, as their C7 packaged daemons are all configured to use systemd.

  • Exactly what John said.

    What I meant by petitioning the board is for a group of people who are willing to actually contribute the time/effort required to make/modify software that would use something other than systemd.

    If such a group exists, and if that group wanted a way to get such software into CentOS, they could petition (ask) for the starting of a SIG.

    If there are not people willing to invest that effort, then we get what we get from the released source code.

    We don’t need people to tell us they don’t like it .. just like we don’t need people to tell us the want they want the upgrade tool to work. We need people who will actually DO SOMETHING to volunteer to do said something in order for these (or any other things) to actually happen.

  • I agree, John.

    We respect CentOS for what it is (blatantly said, being RHEL binary replica, I know it is more…). Debian/devuan is a core system developers team split. You want similar core system developers split here – the place is RedHat, not CentOS and this definitely will not happen (note the difference between debian.org and redhat.com).

    So, the only productive thing about systemd on CentOS list will be to drop all battles, accept systemd as our reality, and keep all posts restricted to pure technical questions/answers.

    Just my $0.02

    Valeri

    ++++++++++++++++++++++++++++++++++++++++
    Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247
    ++++++++++++++++++++++++++++++++++++++++

  • Actually, a LOT of that work exists in Scientific Linux. The power of OSS is that odds are our problems aren’t new and have already been solved by someone and we share those solutions.

    Previous to systemd, the packages didn’t have systemd integrated, so should be good models.

    I, for one, volunteer.

  • Johnny Hughes wrote:

    Yup. I’d happily sign, also… but between work and a *lot* of personal issues, I don’t have the time or energy to do such development. Given that, I wasn’t going to push for it.

    mark

  • I’ll do what I can providing people recognise I’m currently grossly overloaded with community and family responsibilities and currently am lucky if I get 6 hours sleep a day. However I’ll try.

    What is the advantage of patches over a virgin version that can be subsequently patched ?

  • That’s just skimming the surface.

    The real hard bits come from the way systemd hooks into the whole FreeDesktop infrastructure and vice versa. (e.g. dbus is now inextricably part of systemd, and many FreeDesktop interactions happen via dbus.) This is why the BSDs are either dropping GNOME and KDE (e.g. Lumina in TrueOS) or have badly lagging ports compared to the upstream version.

    I suspect it’s probably easier to start with C6, then backport as much as is possible without dragging in any systemd stuff, the same way the BSDs are doing.

    Good luck to y’all. Sincerely. I plan to keep on using C7, warts and all.

  • Doing the change as a patch to the upstream RPM means that, most of the time, you can just apply your patch again whenever the upstream RPM changes. If the patch applies cleanly, chances are that your port update is done, right there.

    If you fork the package base entirely, you have to backport each change from upstream yourself, which is a much bigger burden. Chances are good that you’ll end up forking the whole OS that way, rather than creating a variant spin of it.

    The fork-the-whole-thing model would make sense if you’re starting from C6, since it’s on the downhill side of the patch rate curve now, so that there will be little backporting necessary. And soon, a maintainer of a C6 fork would be on his own anyway, when the upstream patches dry up.

    Whereas if you start with C7, you’d like to have the benefit of the upstream changes while you do the smallest amount of work you can while achieving your end of ridding C7 of systemd. The project is likely to take years to complete — after all, it took many years for systemd to get to where it is now — so there’s a good chance you could complete the project before you’re left with an EOL’d C7 base package set.

    If you look inside many RPMs, they are composed of the untouched upstream source tarball plus a series of patches. One RPM I looked into recently had something like 30 separate patches applied to it, some from Fedora, some from Red Hat, Inc, and potentially (though not in this specific case) some from the CentOS project. This is the normal way of doing these things.

  • Yes, 7 does track upstream. upstream 6 uses systemd also and Scientific Linux 6 does not. I would say that indicates a solution.

  • Uh I’d urge you to recheck your sources as EL6 has never in any part of its lifespan made use of systemd

  • RHEL6 has used Upstart since RHEL 6.0, and continues to use it in RHEL
    6.9. I have no idea where you’d get this kind of information.

  • If you really thought Redhat would switch from upstart of systemd, within a major release, I have no idea why you’d want to use anything based on Redhat.

    jh

  • I think we had enough of Systemd flaming last month. Please stop polluting my inbox and find an operating system compatible with your worldview. It is really tiresome to keep on hearing about it.

  • Huh. Okay, though I’m not sure when you became arbiter of this list. If you don’t like ‘our worldview’ discussions, maybe you need to find a different OS that suits your childish attitude. Like Windows 95.

    Mailing lists now are so full of children it’s hard to even use them.
    Maybe you should leave IT if heated discussions make you uncomfortable.

  • I certainly would not suggest anyone leave this list or stop using CentOS.

    While I don’t think we need to be yelling at each other, I do sympathize with anyone frustrated by the continued ignorance of some of the more vocal proponents of the systemd-haters crowd. The CentOS
    list continues to be a good resource, even if it’s learning about systemd. Sometimes the complaints about systemd can be turned into a learning experience (such as how fstab works). I think that if we can attempt to frame questions about systemd in a more positive way, everyone would get more out of it.

  • Mark Haney wrote:

    Folks, I’m the one who made the original annoyed throwaway remark. I’ve even asked that we end the incipient flamewar. Look, as much as I dislike systemd, going on and on and on just ain’t of interest. Hell, I’ll probably skim and delete, or just delete.

    Now, the information that someone posted about what might be happening to cause my original question was helpful, and in *that* context, in the same email, cmts about systemd, sure. But I dunno ’bout most of you, but a flamewar that runs for *weeks*, as we’ve seen here, is of no interest.

    Maybe we need another mailing list, like alt.religion.editors*, we could have alt.religion.systemd….

    mark

    * vi, not emacs! Nyahhhh….

  • I was sorely tempted to post saying I would initiate an empty email to the list in a week with subject systemd and see what the response would be – I’ll refrain…

    —– Original Message —

  • somehow nobody reads old stuff, or maybe everybody ignores it. Bill Joy and Mark Horton wrote DECADES ago that its name is Vee-Eye, not Vye, or Six (or 6).


    ——————————————————————————-
    Under no circumstances will I ever purchase anything offered to me as
    the result of an unsolicited e-mail message. Nor will I forward chain
    letters, petitions, mass mailings, or virus warnings to large numbers
    of others. This is my contribution to the survival of the online
    community.
    –Roger Ebert, December, 1996
    —————————– The Boulder Pledge —————————–

LEAVE A COMMENT