Disable/mask NetworkManager Leads To Unit Startup Fails
Hello!
For me it was best practice to disable “NetworkManager” on headless installations.
Now suddenly I ran into an problem with several programs not starting correctly upon boot anymore.
The problem seems to be that their unit files contain “After=network.target”
but network.target wont wait till network is up and working, just waits for some low level network stuff.
Now I read that it would be good practice to edit this and use
“After=network-online.target” instead. But I don’t want to touch several systemd unit files shipped via rpm packages.
Software like unbound, postfix and so on which needs to bind to ip+port on startup fails on my systems now since I disabled NetworkManager.
What solved the problem for me was to install “systemd-networkd” instead and enable it upon boot.
It’s system unit file pulls in “network.target” as “Wants” this way software that says “After=network.target” will start after network cards are configured. Why there is this behavior I haven’t found out yet.
Whats your best practice to archieve stable networking upon reboot? Do you use NetworkManager or systemd-networkd or maybe something exotic anyway?
PS: I have already filed an Bugzilla report here:
https://bugzilla.redhat.com/show_bug.cgi?id12570
Thanks!
3 thoughts on - Disable/mask NetworkManager Leads To Unit Startup Fails
Use NetworkManager … it’s the supported RH configuration tool.
This article is a little old (7.3 had a NM rebase and I’ve not revisited it yet … the prime gain in 7.3 though is arbitrary layering of vlan/bridge/bond/etc negating the caveat at the end for edge case).
https://www.hogarthuk.com/?q=node/8
As for the targets … man systemd.special to get a better understanding:
https://www.freedesktop.org/software/systemd/man/systemd.special.html
By default services you install will listen on all interfaces, usually a configuration with something like ListenAddress :: to indicate this
(there is an internal ipv6->ipv4 translation going on so you only should listen on :: and not 0.0.0.0 as well) … when a service binds to a port on all addresses then the address does not need to exist yet
– the kernel just handles it.
If you configure your service to bind on a specific IP though then they way the binding system call works it already needs to exist (by default) which is why with the default configuration if the service tries to start before the IPs are applied (quite likely given parallel startup) you have a nice race condition and a fairly high likelihood of failure.
There’s two ways to handle this – and it depends on the service.
You can order it via after (and potential requires if you want no attempt to start the service if the target fails to activate)
network-online.target (see referenced man page) which won’t be complete until at least one IP is on any interface.
If it’s your own application and you can set the flags on the actual bind then you can use so_freebind on the socket opened, some applications may have this as a compile time or configurable option.
Another option is to enable net.ipv4.ip_nonlocal_bind and net.ipv6.ip_nonlocal_bind which allows the kernel to bind a socket on an IP it doesn’t have (do note the caveat it may break some things).
https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt
If it’s a systemd socket being used by the service then you can enable the freebind option via FreeBind=true
https://www.freedesktop.org/software/systemd/man/systemd.socket.html
Overall you may be interested in this:
https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/
I really wouldn’t use systemd-network on CentOS as a general thing as it’s basically undocumented/unsupported by Red Hat at this time, with NM being their actual supported solution.
https://access.redhat.com/solutions/783533 <-- network Vs NetworkManager https://access.redhat.com/solutions/876183 <-- no systemd alternative for NM in EL7 Hope that helps and explains things!
Thank you very much James for your deep explaining answer!
That cleared everything up for me and yes that helped a lot.
Have a nice day :)
—–Urspr
No problem, glad it was helpful.
Incidentally if there are services you need ordered after network-online.target you don’t need to (and shouldn’t) edit the unit files for the services directly.
Instead you should just provide an override with the additional dependency so that systemd merges the two automatically and so future updates of $package don’t break on changes to the unit file in
/usr/lib/systemd/system and any changes to the unit file get picked up for your system automatically, if the route you go down to resolve your use case is via systemd dependency declaration rather than kernel sysctl or application freebind behaviour.
In the case of nginx, for example, you’d do this:
mkdir /etc/systemd/systemd/nginx.service.d cat > /etc/systemd/systemd/nginx.service.d/after-network-online.conf <
This snippet you could drop in for any service that you needed ordering after network-online, ideally via some CM solution like ansible, puppet, chef or saltstack of course.
As an incidental note to this whole thing … nonlocal_bind and freebind are both viable answers and good methods for handling things like postgres, mysql, haproxy and similar where you need them to bind on a specific IP which is a VIP moved by keepalived (or pacemaker or similar tool) and not always existent on a box without having to start the service after IP is moved via an orchestration (eg via pacemaker)
…
Of course *that* need is avoided if you just tell the app to bind on
0.0.0.0 or :: which, as mentioned, doesn’t have these issues and is the default behaviour on a lot of applications, with limiting the connectivity via iptables where it should only be visible on a single interface.
Of course there are occasions where you specifically do need to bind to a specific interface, either to differentiate sshd configs or for other reasons, and that’s when the dependency, sysctl and network socket options are various tools to solve the issue.