How To Stagger Fsck Executions

Home » CentOS » How To Stagger Fsck Executions
CentOS 23 Comments

CentOS 6

Hi All:

Over the weekend I had to reboot one of my systems and got hit with fsck runs on all of the filesystems. I would not mind so much except doing them all at once took over an hour. I would like to be able to stagger these, ideally only execute one fsck per reboot. I have been able to think of two possible solutions but neither is terrific.

My first idea was to manually run fsck on each filesystem, one every couple of weeks. That way they will not all come due at the same time if we reboot on a regular basis.

The second idea was to set each filesystem to a different random count value. This would run the risk of having two or more executions at the same time but it would probably not be very frequent.

Does anyone have a suggestion for a better way of doing this?

TIA

Regards, Hugh

23 thoughts on - How To Stagger Fsck Executions

  • Take a look at ‘man tune2fs’ and ‘man fstab’ for modifying the fsck order in your system.

    HTH,
    — Arun Khan

  • Using “tune2fs -c”, set the max-mount-counts to a different prime number for each filesystem. So e.g. instead of having 20 for all of them, set the first filesystem to 17, the second to 19, the third to 23, and the fourth to 29. This way, three or more fscks on the same boot are quite unlikely.

    HTH,

    Kay

  • From: Arun Khan Sent: April 20, 2015 23:49

    Thanks but I did look at those and I was not able to find anything that would limit the fsck executions to one per reboot. Changing the order of execution will not address my concern.

    Regards, Hugh

  • From: John R Pierce Sent: April 20, 2015 23:58

    Thanks. That would avoid the problem on future systems.

    Regards, Hugh

  • From: Kay Diederichs Sent: April 21, 2015 03:43

    Thanks but that is not much different then my second idea and does not fully avoid the problem.

    Regards, Hugh

  • From: Mark Milhollan Sent: April 21, 2015 05:35

    Thanks but changing the order of execution or executing them in parallel does not help with executing them one per reboot.

    Regards, Hugh

  • From: Hugh E Cruickshank Sent: April 20, 2015 21:09

    I have come up with a third idea that would seem to address what I am looking for…

    1. Create a file with the list of filesystems in desired order of
    execution.

    2. Create an init.d script that:

    a. Sets tune2fs -c 0 on all filesystems.
    b. Extracts the first filesystem from the file,
    c. Sets tune2fs -c 1 on it, and
    d. moves it to the end of the file.

    The result is that on each reboot (after the first) only one filesystem will be checked on each boot. The down side is that an fsck will be run on every reboot however this could be mitigated by adding a boot count that would be maintained and checked in the file.

    It would appear that I have the beginnings of a workable solution.

    Thanks for everyone’s suggestions.

    Regards, Hugh

  • Why do you care about running them at the same time when it doesn’t take longer to run them all in parallel? Except I think the root filesystem normally runs first. So you might want to stagger it vs. everything else.

    And unless you reboot frequently you are probably hitting the time setting, not the mount count.

  • How frequently does one reboot (CentOS) Linux? Well, my observation is:
    every 30-45 days there is either kernel or glibc update so you have to reboot. This makes it about 10 reboots a year, so you are pretty much close to hitting mount count as much as time from last fsck for ext[2,3,4].

    As it was already mentioned: XFS is marvellous. I use it forever for huge filesystems on Linux boxes. I remember howto by Russel Ingram was titled
    “Linux + XFS HOWTO. Linux on Steroids”…

    Valeri

    ++++++++++++++++++++++++++++++++++++++++
    Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247
    ++++++++++++++++++++++++++++++++++++++++

  • From: Les Mikesell Sent: April 21, 2015 09:19

    I am trying to avoid running them at the same time in an effort to avoid 70 minute boot times (which is what happened on the weekend). I accept that fscks are required on a periodic basis and I am willing to reboot more often to achieve these but I would like to minimize downtime (during the reboot) where possible.

    This is in fact what transpired on the weekend and I would leave this in place as a protective measure.

    Regards, Hugh

  • How many filesystems do you have? If you look at ./etc.fstab, everything where the final number is ‘1’ (normally just the root filesystem) should complete first, then everything with a 2 will run at once. If the other mounts are each on different drive/spindles they won’t conflict with each other and will complete in the same time as running just the largest one of them. If you are running fscks of partitions on the same drive in parallel it will obviously go slower.

  • Why do you accept that? The default behavior for filesystems set up by Red Hat tools (anaconda) is not to fsck. Not by mount count, nor by time. The default behavior for e2fsprogs was changed to disable periodic fsck in Feb 2011. CentOS 6 includes a version of e2fsprogs from before that change, but the filesystem is considered very stable, and the periodic fsck is not generally considered necessary.

  • From: Les Mikesell Sent: April 21, 2015 09:54

    It varies from system to system but is typically 8-10.

    I am aware of that. With the exception of /, /boot and /home which are on one spindle (actually a hardware mirrored pair) the remaining filesystems are on separate drives (actually hardware mirrored pairs or RAID 10 arrays). The largest of the filesystems (four of them) share a common SAS controller, data channel and external disk array hardware
    (HP D2600) so running these in parallel might not be as effective as they could be.

    Regards, Hugh

  • From: Gordon Messmer Sent: April 21, 2015 10:30

    Every article I have read on the subject has recommended this a good practice.

    I have confirmed that filesystems setup by anaconda on both CentOS 6
    and RHEL 6 have both boot count and interval disabled however they are not disabled for any manually created filesystems (they are set to 24 and 6 months, respectively).

    I find it interesting that as late as 2014 Red Hat is recommending:

    . If automatic filesystem checks are inconvenient, then it is
    recommended to disable the automated filesystem check as discussed
    in the following article:

    How to turn off forced/automatic fsck in Red Hat Enterprise Linux?

    . Once disabled, it is recommended to schedule regular “human
    controlled/monitored” filsystem checks, when it is convenient to
    do so. These checks should not be ignored, or scheduled too far
    apart.

    This is from https://access.redhat.com/solutions/70531

    Regards, Hugh

  • Thus converting “Enterprise” into more or less home user system – for user with plenty of time to spare for manual system maintenance… ;-(

    Valeri

    ++++++++++++++++++++++++++++++++++++++++
    Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247
    ++++++++++++++++++++++++++++++++++++++++

  • You may be missing a key fact of how prime numbers work.

    You can only get two or more fscks on a single reboot when the mount count is a multiple of two or more of the max-mount-count values. When those numbers are all prime, the frequency of such occurrences is much lower than when you use purely random values.

    With the four values that Kay provided, I calculate a 1.2% chance on average that two or more volumes will need to be checked on the same reboot. If you reboot on average once a month, that means it only happens once every 7 years or so. That means the machine may well be retired before it happens even once, and then only if you reboot that often in the first place.

    If you take the same set of values and add one to them to make them all even, and thus composite numbers, the chance rises to 3.3%, or about once every 2.5 years for monthly reboots. Thus, Kay’s solution is actually more than twice as good as your second solution.

    Interestingly, the numbers don’t actually have to be prime to take advantage of this property of integers. They just have to be *relatively* prime: http://goo.gl/bQbu5Z

    For example, the set {15, 19, 22, 23} isn’t all prime, but is *relatively* prime, since there is no integer other than 1 that evenly divides any pair among the set. This set gives about a 1.5% chance of a 2+ volume collision, nearly as good as Kay’s prime set.

    I’ve put the calculator I built to test this up here:

    https://gist.github.com/wyoung/7c94967bb635de48d058

  • Ooops, forgot to mention one other minor detail:

    This calculator gives the chance or a 2+ volume simultaneous check, which is actually a much more stringent case than you ran into. If you have four volumes and are using a relatively prime max-mount-count set, you won’t get a simultaneous check of all four volumes until your reboot count equals the *product* of all values in the set.

    With Kay’s prime set, that means it won’t happen until you have rebooted 215,441 times. If your machine lives 10 years, that’s enough to allow for about 60 reboots a day while still having *zero* chance of a 4-volume fsck.

    I think it’s fair to say that Kay’s solution (or a relatively-prime set) does actually solve the problem.

  • Not every source is equal.

    The maintainers turned that behavior off by default sometime around the release of RHEL 6 (before the release, but after the package set was finalized). Among at least any group I am aware of, ext had been considered very stable for quite a long time before that, and common practice was to disable periodic checks. (As Anaconda has since who-even-knows when. WAY back.)

    Right. Red Hat’s policy was to disable the checks long before that default changed in e2fsprogs (in version 1.42).

    CYA?

    I’m not aware of such a recommendation in any of their publicly available documentation.

    Can’t access it, myself. My university has a site license for RHEL, but it doesn’t give me access to the KB.

  • From: Warren Young Sent: April 21, 2015 14:13

    You are right and I stand corrected.

    I have done some “what if” testing. Assuming 8 filesystems and weekly reboots over a ten year period…

    The random numbers would result in as many as 40 weeks per year where
    2, 3 or 4 fscks would be run in the same week depending on the random numbers selected.

    Using the prime numbers 7, 11, 13, 17, 19, 23, 29 and 31 the is a maximum of 7 incidents per year of 2 fscks per week and none for 3 or more.

    Clearly the prime numbers are better.

    Thank, Hugh

  • Using which tool? My simulator, or something you cooked up yourself? If the latter, would you care to share?

    I’ve updated mine to break out the stats for 3+ volumes instead of just reporting all multi-volume fscks together:

    https://gist.github.com/wyoung/7c94967bb635de48d058

    Then I rewrote that in C++, since these 8-volume simulations were literally going to take days with the Perl version:

    https://gist.github.com/wyoung/966383b4efbb63aafc71

    (The Perl version is about 1/5 the speed of the C++ one. This actually isn’t all that bad, considering that it’s an interpreted dynamically-typed language going up against a statically-typed native-code compiled language.)

    This is why I pointed out that you only need *relatively* prime numbers: so that if you decide the largest max-mount-count can’t be over 31, you don’t have to go clear down to 7 in order to find the last prime for 8 volumes.

    Using relatively-prime numbers, you can skew the set upwards quite a bit without increasing the largest value. The most efficient set I’ve been able to come up with is:

    17, 19, 23, 25, 27, 28, 29, 31

    The three composite values (25, 27, and 28) do not share any common factors: 25 uses 5 twice, 27 uses 3 thrice, and 28 uses 7 plus twice 2.

    My newer simulators give these results for the chances of a multi-volume fsck with your prime set:

    period: ~6.7 billion
    2-volume: 8.12%
    3-volume: 1.08%
    4-volume: 0.09%
    >5: < 0.01% chance total: 9.3% My relatively prime improved set gives these results because the set’s median is higher while keeping the same maximum, while also avoiding any reuse of prime factors: period: ~126.2 billion 2-volume: 0.37% 3-volume: 0.33% 4-volume: 0.02% >5: < 0.01% chance total: 0.7% See? Number theory *is* useful in real life. :)

  • From: Warren Young Sent: April 22, 2015 20:46

    I cobbled something together in OpenEdge ABL. I have uploaded it to

    http://pastebin.ca/2979494

    This was intended only for my use so, while the code is relatively clean, it is not documented.

    Regards, Hugh

  • Thanks!

    I like how it saves work by taking real world conditions — service life of the machine and reboot frequency — into account. Mine just computes all of the possibilities, even if in practice it would require a machine a million years old to hit them all.