Bypassing ‘A Stop Job Is Running’ When Rebooting CentOS 7
I’m currently trying to reboot a CentOS 7.5 workstation (to complete an upgrade to 7.6), but it is ‘stuck’ while shutting down with ‘A stop job is running for …’ – the counter initially gave a limit of ‘1min 30s’ –
but each time it reaches that limit, it just adds on ~90 seconds to the limit …
Currently the limit is ’25min 33s’
I’m in no hurry to have this workstation operational, but I guess at some point I will have to power cycle it …
Does anyone know how to bypass this? – or at least stop it increasing the limit each time it is reached?
It does seems rather pointless to keep increasing the limit like this …
Thanks
James Pearson
10 thoughts on - Bypassing ‘A Stop Job Is Running’ When Rebooting CentOS 7
James Pearson wrote:
It _finally_ gave up at 30 mins and rebooted
James Pearson
James Pearson wrote:
One question: did it have a mounted nfs filesystem?
The joys of systemd….
mark
mark
Yes, NFS integration with systemd is broken by default, at least it was still the case when I last checked. If you want it to work correctly, you have to add
‘x-systemd.requires=network-online.target’ as NFS mount option.
Clearly, how should systemd know that NFS won’t work without network? I
knew you agree :-)
Regards, Simon
“Anything Windows can do, systemd can do better” (with apologies to Irving Berlin).
mark wrote:
All our boxes have NFS mounted files systems – and usually this isn’t a problem – reboots work without an issue
In this case, it appeared to be ‘stuck’ on a local file bind mounted over a file on an NFS mounted file system
But that isn’t really the point – I don’t really want to have to wait a maximum of 30 minutes for the reboot to give up waiting for ‘whatever’
Poking about a bit, I see that /usr/lib/systemd/system/reboot.target has the line:
JobTimeoutSec0min
(there is a similar JobTimeoutSec0min in poweroff.target)
I’m guessing I could create something like
/etc/systemd/system/reboot.target.d/override.conf containing something like:
[Unit]
JobTimeoutSec=3min
Now I need to see if I can reproduce the issue and see if this setting works …
James Pearson
I’m not sure it’s right to blame systemd. Systemd asked nicely for the service to shutdown. The service didn’t, probably because the update change something and pulled the rug out from beneath it. Systemd then waited a bit to make sure the service wasn’t just being slow, and finally gave up and forcibly killed it. I think this is a reasonable approach to killing a misbehaving service while trying to minimize data-loss, and the timeout can be configured.
This hasn’t happened to me recently, but I think I’ve tried Ctl-C and Ctl-Alt-Del without much success. That leaves the Big Red Switch
(which is mostly small and black these days).
Jim
But we can blame systemd for the cryptic message
A stop job is running
Surely systemd knows what service it is waiting for, why doesn’t it tell us?
The stop job XYZ is running
jon
I didn’t read this thread all that carefully, but has anyone mentioned editing /etc/systemd/system.conf and changing DefaultTimeoutStartSec and DefaultTimeoutStopSec to a lower value?
Scott Robbins wrote:
All the entries in /etc/systemd/system.conf are commented out – and as the systemd-system.conf man page states:
By default the configuration file in /etc/systemd/ contains
commented out entries showing the defaults as a guide to the
administrator
The DefaultTimeoutStopSec line in /etc/systemd/system.conf is:
#DefaultTimeoutStartSec
The message reported by the OP and the message I see is ‘A stop job is running for …’ where the ellipses stand in for the unit that systemd is waiting for. It seems pretty clear IMHO. Actually debugging it is harder since the system is not available during shutdown, but that’s a generic problem and the systemd docs do provide debugging tips.
Jim