Ulimit Problem – CentOS 5.10

Home » CentOS » Ulimit Problem – CentOS 5.10
CentOS 10 Comments

Running across some curious stuff with ulimit on CentOS 5.10.

We have a non CentOS packaged version of Asterisk (using their packages) that we start at boot time with a typical RC script.

Recently it started whining that it couldn’t open enough file handles.

As we dug further into this, it appears that at boot time, it inherits ulimit from init, which is pretty low: 1024.

We’ve set /etc/security/limits.conf and sysctl significantly higher both for root/system wide and also for the user Asterisk runs under, to no avail.

If we log in via SSH as root (or sudo) the correct root/system-wide ulimit of 8192 is set.

Looking in /proc/[PID]/limits shows the lower ulimit only if the package is started from init (standard rc3.d type sysV stuff…). If we restart it via console, remote ssh, anything else, the limit bumps to 8196.

Attempting to force the ulimit up inside the RC script has no effect, since the package is running as a non-root user. It fails to raise the limit.

Right now, we’re not totally against just taking it out of the startup and starting it manually anyway, since we really don’t want the Asterisk platform coming up after a crash/reboot anyway, and any other reboots there will always be a human involved, but the way init is handling ulimit seems utterly retarded and broken.

Some indication (different engineer found it, I haven’t seen the RHEL case number) appears to indicate that folks wanted init ulimited heavily in case of startup DDoS type stuff, but we haven’t figured out a semi-sane unix-conventional type way AROUND this when it’s needed that if we were hit by the proverbial bus, a “normal” unix guy would find.

Perhaps we’re missing something.

Thoughts?

10 thoughts on - Ulimit Problem – CentOS 5.10

  • Yep, it’s rather low.

    I’ve had success making changes in limits.conf (for MySQL in my case). In my case I increased both the system wide and per user (mysql user).

    Example:
    hard nofile 2048
    soft nofile 2048

    SSH to that box (as ) … run ulimit -Hn and -Sn … bingo 2048

    Are you able to switch users to the user your Asterisk user to run `ulimit
    -Hn` and `ulimit -Sn`?
    Once you’ve verified new sessions get the proper limits, stop and then start your Asterisk daemon. Hopefully from there it will stop whining about file handles.

    Interesting – I’ve not seen that odd behavior. It worked across the board
    (initscript and all) when I made changes via limits.conf.

  • Nathan Duehr wrote:

    init.d scripts run as root so you should be able to set a hard/soft limit in the init.d script before the package is started

    James Pearson

  • Am 24.04.2014 um 19:44 schrieb James Pearson :

    to survive updates, the init script should include one config file under /etc/sysconfig/ – there should such a entry be safe.

  • Tried that. Didn’t work. Very interesting.

    Also attempted /etc/security/limits.conf for both “*” and the asterisk user, to no avail. It’s not honored by init-started stuff, apparently on CentOS 5.10.

  • This is definitely true. The init script came from the Asterisk folks (we’re not using a package modified to behave specifically with RHEL/CentOS) and would get over-written on upgrade.

    Seems like the brokenness is the behavior of init ignoring /etc/security/limits.conf, to my way of thinking anyway.

  • Umm, no. That’s you not understanding what limits.conf is.

    Limits are hard to grok. I had to write a massive document at work explaining it. And people still don’t get it.

    Basically:
    init scripts inherit from init (pid 1), which gets defaults from
    the kernel

    Processes initiated by a user will inherit limits from the the user’s
    environment. For most users that will have involved a PAM session,
    and most PAM configs call pam_limits and _that_ reads limits.conf.

    Doing a ‘su’ will involve PAM and that may cause pam_limits (and thus
    limits.conf) to be read.

    Remember that init processes started at boot time will run as root and so can increase limits. You need to increase hard limits before you increase soft limits.

    Processes started as a user can _not_ increase hard limits. You need to
    “su” to root, or “su” to a user defined in limits.conf to change those values.

    Bottom line: limits.conf is a PAM config setting for pam_limits. It’s not in the general path. Other process _may_ use the file but they need to have root level privs to obey it properly.

  • Am 29.04.2014 um 00:18 schrieb Nathan Duehr :

    Could you show us how do you tried that? Break your task down please.

    __
    LF

  • Actually, I completely understand, but I still had to step through it in various forms and write myself a flowchart. It’s beyond non-intuitive at this point in Linux “design” history.

    For starters, how Linux is handling it is *really* FUBAR compared to some of the commericial *nix variants, but that’s kinda normal… Linux loves to be obtuse and overly-complex at times. It’s an endearing quality. :-) Note that if you had to “write a massive document” it’s probably overly-complex for no good design reason.

    Not processes started that change to a non-root user from a root/init/rc script. No session. At least not from what I was seeing in 5.10. Intended or not, it wasn’t behaving like PAM was ever involved. :-)

    This is all true, but doesn’t apply to the scenario presented.

    The PAM stuff is entertaining… since most new admins would think to themselves, “I’ll just become that user and see what that user’s ulimit is…” do an “su – [user]” and see something and assume it’s what would happen to the user’s limits inside that start script.

    Of course, the best place to look in the end, is /proc/[PID] …

    Here’s the real fun… the asterisk guys (at least back in this old version), punted… LOL… from their safe_asterisk secondary start script that gets called from their main init.d script…

    # since we’re going to change priority and open files limits, we need to be
    # root. if running asterisk as other users, pass that to asterisk on the command
    # line.
    # if we’re not root, fall back to standard everything. if test `id -u` != 0 ; then
    echo “Oops. I’m not root. Falling back to standard prio and file max.” >&2
    echo “This is NOT suitable for large systems.” >&2
    PRIORITY=0

    Even more fun… if your non-Production environment is KVM virtuals, and your Production systems are on bare-metal… don’t get me started on that one… but that’s a pretty common scenario these days, too. (Limits are exceedingly horked on KVM VMs… haha…)

    Anyway, we’re all sorted here, and I had a hell week doing a lot of other stuff in-between, but it was a fun reminder of the whole ulimit mess. I suspect this stuff will be even more entertaining as init goes away in lieu of upstart and similar. Heh.

    Appreciate the thought put into the reply. Perhaps someone struggling with this down the road will find the thread via Google or similar and realize they have to think a lot harder than really should be necessary to get it set where they want it. Most “basics” type docs out there just point newbies to limits.conf and do all examples from a user’s shell that they got to from su… which isn’t how much really starts/runs on Production servers. :-)

  • If you’re doing it as “su user” then pam.d/su is called which calls system-auth which calls pam_limits. If you’re doing it as “runuser”
    then pam.d/runuser is called which directly calls pam_limits

    If your program just does setreuid() calls (which it can do if started as root, or is setuid) then it’s not going near PAM and so will inherit the kernel defaults (if started by init) or the user current values
    (if started by a user).

  • Yup… guess which one Asterisk did back in 1.4 ? :-)

    (Yeah it’s ancient… [insert usual operational excuses here].)