16 thoughts on - Persistent Change Of Max_stack_depth

• Could anybody point me in the right direction for setting the kernel parameter, max_stack_depth, to 10240 for database tuning?

  I have currently set it by running ‘ulimit -s 10240’ but this does not survive a reboot.

  Look at the file /etc/security/limits.conf

  For documentation, ‘man limits.conf’

  – Thomas

• Hi Thomas,

  Thanks for the response, I’ve been nosing around that file recently but noted the first two lines;

  #This file sets the resource limits for the users logged in via PAM.
  #It does not affect resource limits of the system services.

  I added these two lines to the end of the file

  * soft stack 12288
  * hard stack 12288

  in an attempt to set the stack depth to 12MB so that I can configure PostgreSQL max_stack_depth = 10MB.

  I rebooted, ulimit -s shows 12288.

  When I restart my service (#It does not affect resource limits of the system services.) becomes apparent.

  Aug 14 16:22:17 db1 pg_ctl[3177]: < 2015-08-14 16:22:17.839 BST >LOG:
  invalid value for parameter “max_stack_depth”: 10240
  Aug 14 16:22:17 db1 pg_ctl[3177]: < 2015-08-14 16:22:17.839 BST >DETAIL:
  “max_stack_depth” must not exceed 7680kB. Aug 14 16:22:17 db1 pg_ctl[3177]: < 2015-08-14 16:22:17.839 BST >HINT:
  Increase the platform’s stack depth limit via “ulimit -s” or local equivalent.

  So, I then run ‘ulimit -s 12288’ and still can’t restart my service.

  How can I increase stack depth for system processes, not just PAM
  authenticated users?

  Thanks in advance,

  Michael

• What CentOS version?

  If this is CentOS 7 then you may need to put the ulimit directives in the service file.

  An example is I needed to increase the NOFILE limit for nfs-secure on a Fedora 20 machine so I set

  LimitNOFILE384

  In /etc/systemd/system/nfs-secure.service

• Michael H wrote:
  
  Well, a quick and dirty hack would be to add that as a script in
  /etc/rd.local (or whatever the appropriate place is for CentOS 7).

  mark

• Jason is probably on the right track here.

  If it’s CentOS 6 stick ‘ulimit -s’ in the init script If it’s CentOS 7 use LimitSTACK= in the service file

  -Thomas

• It’s already been pointed out that you’ll need to modify the init script, but just to make clear why that is the case, I thought it should be pointed out that “ulimit” is a bash built-in command. When you run
  “ulimit -s 10240” you’re not modifying the system, as a whole. You’re only modifying that shell’s environment and any child process that you start from that shell. Not only does the command not survive a reboot, it doesn’t persist between logins.

  And that’s one of the better features of systemd. Because services are always started by systemd, they don’t inherit environment, or process limits, or SELinux context from a login shell. If they start correctly interactively, via systemctl, they’ll start correctly the next time the system boots. Under the old init system, that wasn’t true.

• The problem with it though is it ignores system wide and user specific baseline settings. If systemd still obeyed ulimits.conf I would agree that it was an improvement.

• I suggest putting it in the sysconfig file instead, if such exists.

  /mark

• Sure, but how many init scripts provide for adding an extra command via sysconfig files? Most of them only allows for adding some predefined options to main service start command.

  Thomas

• Am 15.08.2015 um 23:38 schrieb Thomas Eriksson :

  while processing the init script the sysconfig files are sourced and completely parsed as they where part of the init script. So, everything will interpreted.

• Yes, you are absolutely correct, my bad.

  Thomas

• Hi Jason,

  CentOS7.1

  I located the service file for PostgreSQL-9.4

  [root@db1 multi-user.target.wants]# locate PostgreSQL-9.4.service
  /etc/systemd/system/multi-user.target.wants/PostgreSQL-9.4.service
  /usr/lib/systemd/system/PostgreSQL-9.4.service

  I’ve edited
  /etc/systemd/system/multi-user.target.wants/PostgreSQL-9.4.service and added

  LimitSTACK288

  to the [Unit] section of the service file.

  systemctl daemon-reload systemctl restart PostgreSQL-9.4

  I’m still getting the same errors when I try to start my service…

  Aug 17 08:23:58 db1 pg_ctl[16839]: < 2015-08-17 08:23:58.533 BST >LOG:
  invalid value for parameter “max_stack_depth”: 10240
  Aug 17 08:23:58 db1 pg_ctl[16839]: < 2015-08-17 08:23:58.533 BST >DETAIL: “max_stack_depth” must not exceed 7680kB.

  I have tried moving the ‘LimitSTACK288’ to other sections of the service file, when I do I don’t get my error from PostgreSQL I see this in /var/log/messages

  Aug 17 08:38:47 db1 systemd: Reloading. Aug 17 08:38:47 db1 systemd:
  [/usr/lib/systemd/system/dm-event.socket:10] Unknown lvalue
  ‘RemoveOnStop’ in section ‘Socket’
  Aug 17 08:38:47 db1 systemd:
  [/usr/lib/systemd/system/lvm2-lvmetad.socket:9] Unknown lvalue
  ‘RemoveOnStop’ in section ‘Socket’
  Aug 17 08:38:49 db1 systemd: Starting PostgreSQL 9.4 database server… Aug 17 08:38:49 db1 kernel: PostgreSQL94-ch[3021]: segfault at
  7ffcd1a28f30 ip 00007f116054c79e sp 00007ffcd1a28f30 error 6 in libc-2.17.so[7f1160458000+1b6000]
  Aug 17 08:38:49 db1 systemd: PostgreSQL-9.4.service: control process exited, code=killed status
  Aug 17 08:38:49 db1 systemd: Failed to start PostgreSQL 9.4 database server. Aug 17 08:38:49 db1 systemd: Unit PostgreSQL-9.4.service entered failed state.

  Should I, or can I make this change elsewhere?

  thanks

  Michael

• Just a quick addition –

  Hi All,

  On the PostgreSQL mailing list I’ve been pointed to this URL;
  https://ma.ttias.be/increase-open-files-limit-in-mariadb-on-CentOS-7-with-systemd/

  I’ve created a directory (I tried changing the 9.4 to 9-4 also)

  /etc/systemd/system/PostgreSQL-9.4.service.d
  /etc/systemd/system/PostgreSQL-9-4.service.d

  created a ‘limits.conf’ file inside this directory containing
  [Service]
  LimitSTACK288

  and I’m unable to restart the service still…

  I see the same errors as above when I run systemctl status PostgreSQL-9.4 -l.

  any more suggestions?!

  thanks

  Michael

• Hi All,

  I have created a new directory
  /etc/systemd/system/PostgreSQL-9.4.service.d

  restorecon -Frv /etc/systemd/system/PostgreSQL-9.4.service.d

  I created a limits.conf

  I have tried to add this with no section –
  LimitSTACK288

  output:
  Aug 17 11:18:41 db1 systemd:
  [/usr/lib/systemd/system/dm-event.socket:10] Unknown lvalue
  ‘RemoveOnStop’ in section ‘Socket’
  Aug 17 11:18:41 db1 systemd:
  [/etc/systemd/system/PostgreSQL-9.4.service.d/limits.conf:1] Assignment outside of section. Ignoring. Aug 17 11:18:41 db1 systemd:
  [/usr/lib/systemd/system/lvm2-lvmetad.socket:9] Unknown lvalue
  ‘RemoveOnStop’ in section ‘Socket’

  the [Unit] section –
  [Unit]
  LimitSTACK288

  output:
  Aug 17 11:20:06 db1 systemd:
  [/usr/lib/systemd/system/dm-event.socket:10] Unknown lvalue
  ‘RemoveOnStop’ in section ‘Socket’

  Aug 17 11:20:06 db1 systemd:
  [/etc/systemd/system/PostgreSQL-9.4.service.d/limits.conf:2] Unknown lvalue ‘LimitSTACK’ in section ‘Unit’

  Aug 17 11:20:06 db1 systemd:
  [/usr/lib/systemd/system/lvm2-lvmetad.socket:9] Unknown lvalue
  ‘RemoveOnStop’ in section ‘Socket’

  the [Service] section –
  [Service]
  LimitSTACK288

  output:
  Aug 17 11:21:55 db1 systemd:
  [/usr/lib/systemd/system/dm-event.socket:10] Unknown lvalue
  ‘RemoveOnStop’ in section ‘Socket’
  Aug 17 11:21:55 db1 systemd:
  [/usr/lib/systemd/system/lvm2-lvmetad.socket:9] Unknown lvalue
  ‘RemoveOnStop’ in section ‘Socket’

  and the [Install] section –
  [Install]
  LimitSTACK288

  output:
  Aug 17 11:23:23 db1 systemd: Reloading. Aug 17 11:23:23 db1 systemd:
  [/usr/lib/systemd/system/dm-event.socket:10] Unknown lvalue
  ‘RemoveOnStop’ in section ‘Socket’
  Aug 17 11:23:23 db1 systemd:
  [/etc/systemd/system/PostgreSQL-9.4.service.d/limits.conf:2] Unknown lvalue ‘LimitSTACK’ in section ‘Install’
  Aug 17 11:23:23 db1 systemd:
  [/usr/lib/systemd/system/lvm2-lvmetad.socket:9] Unknown lvalue
  ‘RemoveOnStop’ in section ‘Socket’

  By the errors I will assume that it should be in the [Service] section. I couldn’t find confirmation of this online…

  When I start PostgreSQL-9.4

  Aug 17 11:25:15 db1 systemd: Starting PostgreSQL 9.4 database server… Aug 17 11:25:15 db1 kernel: PostgreSQL94-ch[3762]: segfault at
  7fffe1f35380 ip 00007f9ccebec79e sp 00007fffe1f35380 error 6 in libc-2.17.so[7f9cceaf8000+1b6000]
  Aug 17 11:25:15 db1 systemd: PostgreSQL-9.4.service: control process exited, code=killed status

  Jason, you mentioned that you have made this change for nfs-secure, I
  tried making the changes directly to the service file but received all of the above errors, I followed the instructions at the to of the file, created

  /etc/systemd/system/PostgreSQL-9.4.service containing

  .include /lib/systemd/system/PostgreSQL-9.4.service
  [Service]
  LimitSTACK288

  and I still see the same errors

  Aug 17 11:31:39 db1 kernel: PostgreSQL94-ch[3800]: segfault at
  7ffda4ca67d0 ip 00007fe592ef479e sp 00007ffda4ca67d0 error 6 in libc-2.17.so[7fe592e00000+1b6000]
  Aug 17 11:31:39 db1 systemd: PostgreSQL-9.4.service: control process exited, code=killed status

  Can anybody else make any suggestions?

  thanks in advance,

  Michael

• …

  Yes, it belongs in the [Service] section.

  $ man systemd.exec
  …
  “The execution specific configuration options are configured in the
  [Service], [Socket], [Mount], or [Swap] sections, depending on the unit type.”

  However, I assume that you are confused because “ulimit” in a bash shell returns a value in KiB, but LimitSTACK and setrlimit accept a value in bytes. That is, you’ve decreased the stack size to 12KiB, which is why PostgreSQL segfaults immediately.

  # cat /etc/systemd/system/PostgreSQL.service
  .include /lib/systemd/system/PostgreSQL.service
  [Service]
  LimitSTACK582912

  # grep stack /var/lib/pgsql/data/PostgreSQL.conf max_stack_depth = 10MB # min 100kB

  # systemctl daemon-reload
  # systemctl restart PostgreSQL

• Hi Gordon,

  That is the most valuable piece of information! I couldn’t find this documented anywhere, maybe I just looked in the wrong place…

  Thank you for your help,

  Michael