Slow Authentication On C7

Home » CentOS » Slow Authentication On C7
CentOS 14 Comments

Recently i’ve migrated our SVN server (virtual machine) from C6 to C7
(more precisely – migrated data to freshly installed virtual machine). And we have problem with very slow authentication. Server is configured with SSSD, user data are fetching from our LDAP server. SVN is configured with apache (pwauth for authentication + LDAP search for Require ldap-group).

It takes pwauth even 10 seconds to authenticate. Whet it comes to svn’s externals it could take as long as 9 minutes to _svn up_ project (when there are no commits to fetch). Every external may take even 15 seconds
(and sometimes even more).

SSSD was configured at first with authconfig / authconfig-tui. I was struggling with SSSD configuration but with no success. I’m not sure where to look (SSSD, apache?). How can i debug this issue?

sssd.conf:
[domain/default]
autofs_provider = ldap cache_credentials = True krb5_realm = #
ldap_search_base = ou=Main,o=company id_provider = ldap auth_provider = ldap chpass_provider = ldap ldap_uri = ldap://ldap.our.domain/
ldap_group_search_base = ou=Group,ou=Main,o=company ldap_user_search_base = ou=People,ou=Main,o=company ldap_id_use_start_tls = False ldap_tls_cacertdir = /etc/openLDAP/cacerts ldap_tls_reqcert = allow
#debug_level = 4
refresh_expired_interval = 120
enumerate = True ldap_referrals = False
[sssd]
services = nss, pam, autofs config_file_version = 2

domains = default
[nss]
homedir_substring = /home entry_cache_timeout = 5400

[pam]
pam_id_timeout

apache:
LDAPCacheTTL 30

ErrorLog logs/svn_http_error_log
CustomLog logs/svn_http_access_log “%t %u %{SVN-ACTION}e” env=SVN-ACTION
ServerName svn.our.domain
DirectoryIndex none
DefineExternalAuth pwauth pipe /usr/bin/pwauth
#AddExternalGroup unixgroup /usr/sbin/unixgroup
#SetExternalGroupMethod unixgroup environment


SVNPathAuthz off
DAV svn
SVNPath /home/repos/subversion_free_avr

AuthBasicAuthoritative off
AuthBasicProvider socache external
AuthExternal pwauth
AuthnCacheProvideFor external
AuthType Basic
AuthName “Subversion repository”
AuthLDAPURL ldap://ldap.our.domain/ou=Main,o=company
AuthLDAPGroupAttribute memberUid
AuthLDAPGroupAttributeIsDN off
Require ldap-group cn=programmers,ou=group,ou=main,o=company
#GroupExternal unixgroup
#Require group programmers
#Require valid-user
#AuthzSVNAccessFile /home/repos/svn.access

On same server we have redmine (with database on separate server and LDAP auth) and git repositories (with gitbucket as frontend, also LDAP
auth) but those repos aren’t extensively used right now. Redmine works not-so-bad, so i guess it is not overall server performance issue.

Disks performance (measured under normal workload):
[root@luah pam.d]# hdparm -tT /dev/vda

/dev/vda: (system)
Timing cached reads: 11412 MB in 2.00 seconds = 5710.28 MB/sec Timing buffered disk reads: 522 MB in 3.63 seconds = 143.79 MB/sec
[root@luah pam.d]# hdparm -tT /dev/vdd

/dev/vdd: (/home where all data resides)
Timing cached reads: 10020 MB in 2.00 seconds = 5013.17 MB/sec Timing buffered disk reads: 172 MB in 3.20 seconds = 53.73 MB/sec

It’s comparable with other our VMs.

Any ideas?

14 thoughts on - Slow Authentication On C7

  • W dniu 11.04.2016 o 20:07, Scott Robbins pisze:

    In nsswitch.conf i have:
    passwd: files sss shadow: files sss group: files sss

    DNS works fine. I think that sssd communicates with LDAP server with every authentication – i have tons of following entries in log:

    http://pastebin.com/rZVjk0gW

    And it repeats for same user over and over again. Is this correct behavior?

  • W dniu 12.04.2016 o 09:45, Marcin Trendota pisze:

    authconfig –enableforcelegacy –disablesssd –disablesssdauth –updateall

    Did the trick – so it looks like it’s problem with SSSD. I would like not to remove it, but it’s production server and must works flawlessly.

  • RedHat never really mastered LDAP, unfortunately. I have a by now ancient article, that mentions it.

    What I found back then is that if it EVER uses LDAP for anything, upon boot, it will look for an LDAP server. It doesn’t even have to be one that the workstation uses–it just wants to know there’s one on the network.

    In other words, I have server1 which workstation1 can use for authentication, and server2, which does, say, address book duty.

    The workstation also does authenticate for some things against the server1. If server1 is off, although there’s a local account on workstation, it will look and hang for awhile, till it gives up and finally logs on the local account.

    However, if server2 is on–that is, as long as there’s an LDAP server that it can see, even though that server isn’t used for authentication, workstation1 will, IIRC, try it, see it can’t authenticate, and then use local authentication.

    Again, this is something that happened a long time ago and I haven’t tried in a long while. However, that’s why I asked if LDAP was involved, because it might be the issue.

  • <http://pastebin.com/rZVjk0gW>/ <http://pastebin.com/rZVjk0gW>rZVjk0gW
    <http://pastebin.com/rZVjk0gW>
    behavior?

    What utter nonsense. Just because you poorly configured your system does not mean that Red Hat never really mastered it… And translating very old experiences to CentOS 7 is even more ridiculous and counter productive.

    To the OP enumerate is always painful, I’d remove that for a start.

    My experience with the DAV SVN though is that clients are horrible in their requests… So many it hits it so hard…

    After various testing I ended up going with the Apache LDAP cache module and doing the auth at the Apache level, not system.

    Was far better in performance with the SVN server being hit fairly hard. I
    can try and dig out an example configuration if you would like.

    The bonus here as well is that svn users are separated cleanly from system users… No reason for a dev to have a shell account on there ;)

  • This was my experience too, for what it’s worth. When I first set up a new system pointed at LDAP it was absurdly slow to authenticate. Setting Enumerate to False in /etc/sssd/sssd.conf made all the difference.

  • James Hogarth wrote:

    I’d be *very* interested in that configuration, if you post it here, or offlist, to me.

    mark

  • Me too. Please, post for everyone, or add me to off-list message.

    Valeri

    ++++++++++++++++++++++++++++++++++++++++
    Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247
    ++++++++++++++++++++++++++++++++++++++++

  • Hello,

    I had similar problem recently with CentOS6 machine, which was in another country and had ~100ms latency to the LDAP server. When I did “id user”, it took around 20 seconds. I did some debugging, and when the user was not a member of additional groups, it was much faster (5 seconds), but still slow. It seems that for each member of a group, the client did a query to the LDAP server. I put “ignore_group_members = true” in sssd.conf and now it’s much faster. Can you try this?

    Regards,

  • In my particular case the server is already widely used so I’m not in a good position to test it. But next time I have to set up a new system that authenticates against LDAP, I’ll be sure to do that!

  • The CA.crt assumes that is used to sign the LDAPS certs … replace as required ;)
    This assumes multiple SVN repos under /srv/svn/repos This includes a local userfile for any quick hacks or system things that you don’t want to hit LDAP for – can be removed. This also allows fallback from one server to another if need be, note that it will need to timeout on the first though. This took a fair chunk of load off of our LDAP server and made checkouts a far more pleasant experience.

    Bonus points if you get your CM to change ordering of LDAP servers between repos (or other web auth) ;)

  • Thanks a lot!

    The moment my svn servers start getting hit big time I will know what to change, thanks to you!

    Valeri

    ++++++++++++++++++++++++++++++++++++++++
    Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247
    ++++++++++++++++++++++++++++++++++++++++

  • W dniu 12.04.2016 o 18:56, David Nelson pisze:

    At the beginning it was off. I’ve turned it on hoping it will eventually speed up authentication. With no success.