Am 01.10.2017 um 17:21 schrieb hw:
how to manage tmpfiles.
Curious, how did you install MariaDB that you have such a problem? The package shipping with CentOS does not create such issue.
You can try chattr?
2017-10-01 12:26 GMT-03:00 Alexander Dalloz :
I think you’ll find that’d do little useful on a tmpfs volume to preserve files across reboots.
What did you put in there that causes such issues in the first place?
There shouldn’t be a thing in that directory except the PID of the running process. It stands to reason that would be emptied on restart since the service would be shutdown cleanly (usually) before rebooting.
If you’ve got stuff in there that a) you need to keep across reboots and/or b) data that requires root access, you’re simply not doing it right. Might I suggest reading up on how CentOS/RHEL’s directory structure and what should go where? It seems to me you’re trying to put a square peg in a round hole and getting frustrated because it won’t fit.
Mark Haney Network Engineer at NeoNova
919-460-3330 option 1
Alexander Dalloz writes:
Thanks, I´ll look into that. I wouldn´t consider a directory like
/var/run/mariadb in any way as only temporary — and wouldn´t consider directories that are required for the system to work as temporary, either.
I´m using the packages from mariadb.org. The old version that comes in CentOS isn´t recommended, and I need features only the newer versions provide.
Lighttpd is from epel, and it has basically the same issue.
“Didn’t work” is an error.
marcos valentine writes:
Wow, I never needed/used that. Being able to make files undeletable might be a very useful thing …
chattr is a valuable but lesser-known tool, if you use it then document it somehow so other admins don’t stumble over it.
—– Original Message —
That directory isn’t temporary. The files almost always are, but not the directories. As I said, whatever it is you’re doing, it’s wrong. I
wouldn’t continue to keep a setup like that as it’s not standard practice to keep data in /var/run that isn’t temporary.
However, you seem to be insistent on doing things contrary to best practices so….. What issue? That the PID is dropped on reboot? What else are you putting in there? I’m beginning to question whether you know what you’re doing or not. Lighttpd doesn’t store any persistent info in
/var/run/ because, like everything else, /var/run isn’t for persistent data.
as others have mentioned, that will not WORK in this case.
The issue here is that /var/run is a tmpfs mount. That means it is NEVER going to survive a reboot. It is a filesystem created in RAM at boot time.
The design of /var/run on CentOS never needs to carry over files after a reboot .. the purpose of /var/run is to hold the PID of a process that starts at boot .. therefore, when you restart, there will be nothing in there as everything is restarting.
The real question here is, what non CentOS thing are you trying to do where you need to redesign the init system.
On CentOS 7 machines, the /run mountpoint (available via symlink as
/var/run) is a temporary filesystem. Try “df -h /run” to see for yourself. That whole directory lives in memory.
Using systemd-tmpfiles is the most reliable method for ensuring your
/run directories are created and given correct perms at boot. The syntax for /etc/tmpfiles.d/*.conf isn’t terribly difficult, and the files there are easy to manage. See the tmpfiles.d(5) man page for details and examples.
Once your file is in place, you can activate it without messing with other temp files:
systemd-tmpfiles –create /etc/tmpfiles.d/your.conf
Paul Heinlein email@example.com
45°38′ N, 122°6′ W
If the mariadb.org package thinks /var/run is persistent, then it’s not intended for CentOS7. If the EPEL package did too, then there could be serious problems with that package. However, I see that it has a /usr/lib/tmpfiles.d/lighttpd.conf, so it is ok.
Mark, Many Non-CentOS originated packages create directories in /var/run as part of the install, and expect them to still exist after a reboot.
They then fail when starting the service because they’re trying to create a PID / Lock file in a directory that no longer exists. This problem has been around ever since /var/run was moved to tmpfs.
Unfortunately, sometimes we have to use packages other than the official CentOS ones, usually as in this case because we need newer versions.
There is a solution that saves /var/run to disk at shutdown and restores it at bootup but I can’t remember what it is.
Those packages have been built poorly.
Yes, and those packages should know how to work with CentOS 7.
Sure, that can be.
There’s no need to do that (and it’s also messy). Instead, if a package needs a directory to exist in /var/run, then create your own config for systemd-tmpfiles, and drop it into /etc/systemd/tmpfiles.d. Work with CentOS 7, instead of fighting with it.
I saw reference to system-tmpfs in Paul’s post so I had a quick look. YUM
doesn’t seem to know about it, but I’m sure Google will help.
Sorry, meant systemd-tmpfiles
On a CentOS 7 system, do:
man systemd-tmpfiles man tmpfiles.d
Those 2 manpages together explain everything you need to know about creating temporary files and directories in /run at boot time. If you search google, you’ll probably end up at online versions of those same man pages.
Sorry, but if you have to use packages that don’t originate from CentOS
and they do that, then I wouldn’t use them. Period. I’d compile from source before I used something configured that way.
Why is it so hard for people to understand that var/run IS NOT
PERSISTENT and was never meant to be? Do they not teach basic Unix concepts anymore? If you think that setup is acceptable, I wouldn’t hire you to water my lawn as you’d likely water the electrical box along with said lawn.
These are VERY VERY basic concepts. Banging a square peg into a round hole, even in a test environment is a good way to get fired and become unemployable. And believe me, word gets around quickly in IT circles.
If you can’t build from source to keep from using non-standard packages, then you really shouldn’t be doing whatever it is you were hired to do.
This is extremely basic arithmetic here. You don’t do surgery with dirty scalpels, you don’t drive without brakes, these are axiomatic just like /var/run isn’t persistent. It’s been that way at least since I was in HS and college in the 80s and very very likely since the early Unix days.
Honestly, I feel bad for your employer if you think this is an acceptable way to get a system working.
There, I’ve said my piece. Call it a flame if you want, truth hurts and ignoring basic rules is a good way to hurt yourself or other people.
This perspective to some extent employs cutting your nose of dispite youre face. Before Packages were introduced, everyone compiled from source. That was a pain, and a long process, especially when you had dependancies that you also had to compile. Packages eased this process but kept the dependancy issue.
Package managers got round (mostly) both the dependancy problem and updating too. The problem with package maintainers not keeping up to date shows that this still isn’t perfect.
However, if you go back to compiling from source then you lose all of these benefits.
Thankfully I do not earn my keep by watering lawns. I do not believe that this is acceptable, but by the same token I have to earn my keep and that involves having working production servers and services.
I have managed to get round this problem in the past through manually doing the same function as systemd-tmpfiles. It is a small price to pay to have a working, (relatively) up to date server.
If you think using non-standard packages that put /persistent/ items in non-persistent locations like /var/run in production environments is far more acceptable than compiling from source because of package management
‘benefits’ then (to me anyway) you’re lazy and dangerous with critical data. My statement still stands. Let me be clear:
THIS. IS. NOT. ACCEPTABLE.
The fact you’d rather bandaid a problem (in production no less) than follow proper standards or compile from source to avoid said bandaid would be a fire-able offense in any IT shop I’ve ever worked at. The fact you find this acceptable means you’re either the only
‘qualified’ (and even that is subject to doubt) person there, or your management is too ignorant to understand the danger. I’m sorry, but in no way is this acceptable for production level servers. I’m sure, if you asked 100 IT people you’d get 100 to agree with me. Being flippant with production servers is never acceptable.
Of course, most people refuse to listen to logic and reason because they are convinced they are right despite evidence (and best practices over
40+ years of Unix) to the contrary.
I’ll end this by saying, I hope the production servers you have don’t provide critical services that could jeopardize the lives of people.
I’d ask who you work for, to make sure I avoid them at all costs, but I’m not sure I’d be told.
Again, denying 40+ years of Unix design and best practices because you’re too lazy to manage compiling from source to avoid denying those practices is truly one of the most astonishing things I’ve ever seen in the 25 years I’ve been in IT.
Then again, maybe I’m old-fashioned when I expect to do something and do it right rather than half-ass it.
The company I work for, and the livelihood of the hundreds of employees depend on my servers. In the 30 years I’ve been in the industry, I’ve never had problems as you’ve described
Don’t know how long you have been working with UNIX but there was no /var/run 40 years ago!
In 30 years you’ve obviously learned nothing about Unix/Linux. I’d be embarrassed to claim that length of IT service and do something as catastrophically stupid as what you’re doing now. Just because it
‘hasn’t been a problem’ doesn’t mean it won’t. Seriously, if it were me, I’d either retire or hire someone better than you with production servers.
You’d think, with your supposed experience, you wouldn’t use the ‘well it’s never happened before’ as a viable reason for doing something.
That’s ignorant, immature and far more dangerous for your organization than I would be happy with as a CEO or Manager. That attitude is never excusable.
This conversation is over. You refuse to listen to literally EVERYONE
ELSE ON THE LIST and therefore not worth anyone else’s time trying to help you. (Especially mine.)
I showed my daughter this thread, she’s a freshman in the Honors College of Engineering at Virginia Tech majoring in Math and CpE, has been using linux since she was old enough to sit at a keyboard and even she was appalled. If that doesn’t tell you something, nothing will.
Do us all a favor and don’t post to the list unless you are willing to listen to rational human beings.
I have not followed this thread since the first few emails… but has anyone suggested the SCL repo? I see mysql 5.6 and 5.7 there.
mark “this is not the flamewar I’m looking for. I’ll move along….”
The simplest solution that comes to mind at 7:30am is simply bind mounting a directory that’s persistent. You’ll still need to define that directory using tmpfiles.d, but…
echo “D /var/run/mariadb 0755 root root -” >
mkdir /var/run-persistent/mariadb -p
echo “/var/run-persistent/mariadb /var/run/mariadb none bind 0 0” >>
Obviously, test this somewhere before you do it to a system you care about. It’s early, and I haven’t. I’m not sure startup ordering will be correct for this sort of thing…
While FHS notes that *files* should be cleared during the boot process, it does not indicate that directories should be, and that is the source of this problem. Packages on recent Fedora, RHEL, or CentOS releases should use tmpfiles.d to define directory structure within /var/run, but that is a unique and recent development.
Please chill out. There is no need to berate users, here.
It will help you to avoid future unpleasant surprises if you take the time to read up on the Hierarchical File System (HFS) and its relation the Filesystem Hierarchy Standard (FHS). The directories /run and
/var/run, which should be the same place on properly configured systems, are solely to be used for run-time data ONLY. The phrase run-time implies ephemeral data that is not preserved between restarts of the service much less reboots.
Well, Linux is not UNIX. And it never was, we used “UNIX-like” system term. There were far too many changed in Linux some of us do not like. So, let’s take today’s Linux (I almost used the word “modern”) for what its is.
Thanks, Gordon, for pointing to that!
Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247
Assuming that your /usr/lib/tmpfiles.d/mariadb.conf file contains:
d /var/run/mariadb 0755 mysql mysql –
the /var/run/mariadb should be recreated on every reboot… if it is not, perhaps either the mysql user or group do not exist. Check
/etc/passwd and /etc/group.
+1 for SCL –
Mark Haney writes:
Well, what am I supposed to do? The socket (or what it was) needs to be put somewhere, and IIRC, it wasn´t my choice to put it there but is a default. With mariadb, there are some defaults you can´t reasonably change because other software expects files where they usually are. And I don´t want to change that, I just want mariadb and lighttpd and other things to start on reboots rather than being broken because someone decided that files/directories they require are to be deleted on reboots before they can start.
IIRC, lighttpd won´t start unless you mess with where it puts its pid file. I think I had to resort to put it into /tmp or something like that because the place where it´s supposed to put it gets deleted on reboots.
I´ve never before had issues like this.
I can’t believe people are still asking this question after being given appropriate advice. So let me repeat it, and don’t ask again unless you’ve read this properly:
1. /var/run is a symlink to /run, which is a tmpfs mounted in RAM.
2. At reboot, /run vanishes, and EVERYTHING that was in it, vanishes with it.
3. For this reason, systemd ships with a utility called systemd-tmpfiles, which is run early in the boot process, to create any appropriate files and directories in /run. Packages that require directories to be present in /run (for keeping PID files or sockets), should ship with the appropriate tmpfiles.d snippets to have these directories created for them on boot.
4. Finally, if you as a sysadmin are using a package from a repo that isn’t CentOS or EPEL, and this package is not following the CentOS
packaging protocol for data in /run, then it is YOUR own responsibility to fix the package, or create your own tmpfiles.d snippet to create the required directories.
5. Learn about systemd-tmpfiles by reading the man pages of
“systemd-tmpfiles” and “tmpfiles.d”.
This is as clear as crystal. If, despite this instruction, you cannot, or do not want to work with CentOS as it was intended, then stop whining about things here.
*Thank you for the explanation .*
2017-10-05 7:00 GMT-03:00 Anand Buddhdev :
CentOS mailing list CentOS@CentOS.org https://lists.CentOS.org/mailman/listinfo/CentOS
It’s quite obvious you aren’t using CentOS packages. If you refuse to do as best practices insist (and have for nearly HALF A CENTURY) then no one here can help you. It seems to me that 1) you’d be better off compiling from source for your environment, or 2) that you need to follow practices established (probably) before you were born or 3) that you stop asking the list for thing no one in their right mind would do.
How hard is that math?
Mark Haney Network Engineer at NeoNova
919-460-3330 < (919)%20460-3330> (opt 1) • firstname.lastname@example.org http://www.neonova.net < https://neonova.net/>
< https://www.facebook.com/NeoNovaNNS/> < https://twitter.com/NeoNova_NNS>
Harold Toms writes:
Thanks! There is no such file.
Apparently we are not supposed to create such files, and now I must compile mariadb from source because otherwise I wouldn´t be following best practises an be out of my right mind … ;)
Somehow, I doubt I would have a file like that if I did that. I might even get stuck in a compiling loop, following best practises indefinitely …
They do exist.
The server isn´t in production yet, so I can create the file and see what happens when rebooting it. Let´s hope that it doesn´t vanish when I do that …
Please explain how compiling packages yourself turns such packages into standard packages.
It is not acceptable to create the system in such a way that files indiscriminately disappear, without any check whatsoever if that´s ok.
Did they require you to verify all the sources of packages you compiled yourself to make sure that they behave exactly like the packages that come with the distribution?
How is compiling packages yourself not another bandaid?
How is compiling your own packages not a danger?
Then you have to agree that defaulting to using a ramdisk for /var/run
— or anything else — is an utterly stupid idea.
I don´t see how compiling your own packages or randomly disappearing files falls under best practises.
People always make mistakes. Best practises can´t be best practises unless they take this into account, and that involves not deleting, truncating or disappearing files that have been placed somewhere, mistakenly or otherwise, lightheadedly. That particularly applies to unknown files, like files in /var/run, which have been placed there and have not been specified for removal, perhaps due to a mistake of the package manager.
Best practises also involve to generally not delete files unless you can be sure that they can be deleted. That is probably what the FHS
intended by specifying that files in /var/run must be deleted/truncated at boot time, assuming that the programs that created them would do this
(and then create them anew if needed), which can be assumed to be reasonably safe since it implies that unknown files remain.
For all I know, someones life could depend on a file that was placed somewhere mistakenly.
Anand Buddhdev writes:
I haven´t had time to read all of this thread before today.
Lighttpd is from epel.
I´m not whining, and it´s not my fault that someone came up with the extremely stupid idea to use a ramdisk for /var/run. It´s also not my fault that lighttpd appears not to be packaged the way it would need to be, and the same goes for the mariadb packages provided for CentOS by the mariadb people.
Perhaps you should complain to whomever made this change for not waiting until all packages have been modified and to the package managers who didn´t modify them before actually deploying it, for not to mention the stupidity of the idea, rather than accusing me of whining.
Jonathan Billings writes:
Or it is intended for CentOS and not done how it needs to be because CentOS differs from the FHS …
Hm, then how come it´s so troublesome?
I have /usr/lib/tmpfiles.d/httpd.conf. It seems to have remained because I had httpd (which is apache) installed and then switched to lighttpd and removed httpd.
httpd is no longer installed. The only installed package referring to it is apache-commons-logging, and I don´t know why it hasn´t been removed.
The contents of /usr/lib/tmpfiles.d/httpd.conf indicate that this file is for apache. Why hasn´t this file been removed when httpd was removed?
There is no file lighttpd.conf in /usr/lib/tmpfiles.d.
Does this mean that both packages, httpd and lighttpd, have serious problems because they do not remove and do not install files correctly?
Again: lighttpd is from epel.
See : “EPEL packages are usually based on their Fedora counterparts and will never conflict with or replace packages in the base Enterprise Linux distributions.”
If there wasn´t some sort of conflict, then there wouldn´t be an issue with lighttpd.
Mariadb is from the mariadb repo.
It is quite obvious that CentOS causes issues because it is not following the FHS.
I´m not insisting on anything, I´m merely pissed that things are broken. It has turned out that this is due to the way CentOS doesn´t follow the FHS.
Not following the FHS probably doesn´t exactly fall under best practises, but CentOS insists on doing so.
And you guarantee that if I was to compile lighttpd from source, there wouldn´t be any issues? Who says that it is better to compile lighttpd from source rather than using a package specifically made for CentOS
that provides it?
Who says that compiling your own packages falls under best practises? Do you compile all software you´re using yourself because otherwise you´d be refusing best practises?
I thought it needless to say, but compiling from source instead of using packages provided by or for the distribution I´m using is virtually the opposite of following best practises, for a number of reasons. I only do that when there isn´t a better alternative.
Living in the past seldwhen is a good idea. Lots of practises that were established before I was born have disappeared, been replaced by others, or were revised. Why should I follow outdated practises?
It doesn´t compute at all.
Please don´t feel insulted by the following; it is not my intention to insult you. It´s merely what I´m thinking:
I can see you saying repeatedly that things should be like you want them to be because you think that they have been the way you think they are for a long time and that everyone who doesn´t think the same way must be either out of their right mind, or stupid or both. That makes me think that you´re living in the past and, spinning around in that circle, lack the flexibility often times required nowadays, letting aside that there seem to be things you´re mistaken about.
My understanding is also that your way of thinking tries to gain orientation by looking at structures and neglects the contents of those very structures; which is not a good idea because structures without content are hollow and rather meaningless, and anything but seldwhen, the content of these structures is far more relevant than the structures themselves, and requires to adjust ones ways of thinking — and eventually the structures — accordingly for to come up with desirable and adequate results. (IMHO this is something you really need to show your daughter, even if it probably doesn´t compute for you.)
I´m much more understanding than mathematical. Unfortunately, math doesn´t make sense to me, and it is even illogical. That may very well qualify as “being out of my right mind” from your perspective, and I can live with that because from my perspective, it is not true that I am.
This isn´t true, the directory is persistent, and the FHS says:
“Files under this directory must be cleared (removed or truncated as appropriate) at the beginning of the boot process.”
 doesn´t tell you that files in /var/run will disappear at shutdown.
Using a ramdisk to store such files is not compliant with the FHS
because the files are neither truncated, nor removed; they are being disappeared, and not at the beginning of the boot process but at shutdown. Using a ramdisk is not appropriate.
The safe and compliant way would be to truncate the files and not to remove or to disappear them.
The FHS doesn´t say /which/ files should be cleared or removed. I would say that all files the removal or truncation of is not explicitly specified must neither be removed, nor truncated, and that automatically removing files is generally questionable and needs to be done, if at all, with great care.
I can only speculate (and hope) that the intention of the FHS here is that programs creating files under /var/run are supposed to remove or truncate the files they created during the previous runtime, and only those, when the system boots, which would be the time when those programs are being started.
Then how come that the first time I´m seeing an issue like this is only after someone made the utterly stupid decision to use a ramdisk for
It is a change that has been made at some time, and we weren´t told about it. Assuming that people not being informed about a change are stupid because they don´t know about it is a stupid thing to do.
Making things worse by providing dirty scalpels or vehicles without brakes — with or without telling those who are going to use them —
doesn´t make things better, and it can be argued that someone providing those should be fired because of their stupidity.
Alas, the only thing that helps against stupidity is more stupidity. Getting upset about it does not.
Gordon Messmer writes:
That´s a really good point, thank you.
Gary Stainburn writes:
I would also expect that.
A ramdisk? That´s an incredibly stupid idea. Ramdisks should not be used by default.
Is there a way to change that?
That doesn´t protect you in case of a power failure and may increase shutdown durations to a point where the amount of time it takes to shutdown isn´t fully covered by the UPS.
What do you propose as an alternative?
You can test something, like some software, over and over again in any way that comes to mind, and at some point, you may conclude that it seems to be working and may go into production. That conclusion is solely based upon “it hasn´t happened yet”, simply because whatever bug of what you´re testing hasn´t shown any effect yet.
Following your argumentation, you would never have a reason to put something into production, or to use it.
What was she appalled about?
You are right, the EPEL version of lighttpd doesn’t have a tmpfiles.d file. I foolishly expected the Fedora package to be similar to the EPEL package, but looking at the RPM spec file, it only installs the tmpfiles.d file for Fedora 15 or greater. Probably worth filing a bug against EPEL about that.
I can’t explain why your httpd tmpfiles.d file is still there, other than if it was saved as part of an upgrade/removal because of local changes. (and even then, it should have a .rpmsave extension, I
think). It’s not marked as a config file in the RPM, so that’s unlikely.
It is owned by the httpd package:
$ rpm -qf /usr/lib/tmpfiles.d/httpd.conf httpd-2.4.6-67.el7_4.2.x86_64
I think that the important learning points today are:
1.) CentOS7 (and any other distro that uses systemd) will have /run as a tmpfs filesystem, and /var/run points to /run on CentOS7, so even if you think this disagrees with the FHS, that’s the way it is for CentOS.
2.) If you are using non-CentOS packages (and that includes EPEL) to run a service, you might need to create your own tmpfiles.d files. This is just a fact of life now. You can also file bugs against the software telling them that /var/run is ephemeral so don’t store important files there.
3.) Systemd developers aren’t going to sit around and wait for developers to change how they put files. They’ve made a lot of significant changes to how systems are managed, for better or for worst, and as people who maintain CentOS systems, we’ve got to learn to use it appropriately.
There’s a lot of conversations that end, “This is how it’s done.”
“But that’s stupid!” “Well, tough, that’s how it is…” when managing computers, and I’m sure this won’t be the last like it. (I remember early in my career wondering why creat() was spelled that way…)
Am 09.10.2017 um 17:54 schrieb Jonathan Billings:
And fun fact: not only RHEL 7 and thus CentOS 7 does so, but too Debian
9 and Ubuntu 16.04 LTS (I have no newer test install of that distro).
And frankly speaking, I don’t see any indication that this violates with the FHS and that /var/run must persist reboots.
Can we please end this stupid discussion? Enough arguments have been exchanged to make clear that packages are broken if they ignore the fact that /var/run content is ephemeral.
What a strawman. You mean that person would still be alive if the file disappeared during boot rather than shutdown?
Just the opposite, the FHS condones the CentOS arrangement.
Under /var/run it says:
“In general, the requirements for /run shall also apply to /var/run.
It is valid to implement /var/run as a symlink to /run.”
Indeed, there are many UNIX ties Linux had that were severed by hard work of developing systemd and friends. I guess we just have to live with that
Then it’s a big bug, and you should immediately file a bug report for it, so that the packager can fix it. Packages in CentOS as well as EPEL
aren’t perfect, and sometimes need to be fixed. We can help by filing bug reports.
CentOS 7 was released in August 2015, which is over 2 years ago. Any package that hasn’t adapted to CentOS 7’s temporary /var/run by now is badly broken. I would either avoid using it, or file a bug report for it
(and use my own tmpfiles.d file in the meantime).
Or, you can download the SRPM of the package, introduce a tmpfiles.d snippet and rebuild the package yourself.
You have many choices to make it work properly.
I shouldn’t, because I’m not using the package in question. I *have*
used other packages from EPEL, where I’ve seen this problem, and I’ve filed bug reports for them, repackaged them myself, or used my own custom tmpfiles.d file to work around the package’s deficiency.
Stop right there. CentOS *is* following the FHS. Can you please stop this whiny complaint against CentOS, and just accept that the packages you’re using are not properly packaged for CentOS 7?
Then, if you still wish to use them, then apply fixes as I have suggested, and also file bug reports.
You entire basis, by claiming that CentOS is not following the FHS, is wrong. Now stop propagating it.