22 thoughts on - Could Not Complete SSL Handshake To Amazon EC2 Host

• Hi Does the deamon run under xinetd? Then you have to configure the only_from in */etc/**xinetd.d**/**nrpe* to.

  Regards Eric Am 01.05.2015 06:46 schrieb “Tim Dunphy” :

• Hi Eric,

  Thanks for your reply. I do have nrpe running under xinetd on the host I’m trying to monitor.

  And running the nrpe checl locally:

  [root@ops:~] #/usr/local/nagios/libexec/check_nrpe -H localhost NRPE v2.15

  [root@ops:~] #grep only_from /etc/xinetd.d/nrpe
  only_from = 127.0.0.1 216.120.248.126

  And I do have port 5666 open on the security group for this host.

  And I made sure the local firewall was stopped, because I am blocking ports with the security groups instead.

  [root@ops:~] #service iptables status Firewall is stopped.

  It’s only when checking from the monitoring host that nrpe fails:

  [root@monitor1:~] #/usr/local/nagios/libexec/check_nrpe -H ops.jokefire.com CHECK_NRPE: Error – Could not complete SSL handshake.

  Really, really puzzling. This is driving me up a wall!! I hopeI can solve this soon….

  Thanks for any and all help with this one!!
  Tim

• This is strange… Do you have SSL aktive on both systems? Run nrpr localy without parameters
  (this should return some nrpe stats) and check ldd for libssl. Am 01.05.2015 07:32 schrieb “Tim Dunphy” :

• I don’t seem to have that command.

  [root@monitor1:~] #find / -name “*nrpr” 2> /dev/null
  [root@monitor1:~] #

  And that’s on either system.

  And if I do an ldd on both, this is what I can tell:

  Server:

  [root@monitor1:~] #ldd /usr/local/nagios/libexec/check_nrpe
  linux-vdso.so.1 => (0x00007fffd895d000)
  * libssl.so.10 => /lib64/libssl.so.10 (0x00007fc61722a000)*
  * libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fc616e43000)*
  libnsl.so.1 => /lib64/libnsl.so.1 (0x00007fc616c29000)
  libc.so.6 => /lib64/libc.so.6 (0x00007fc616868000)
  libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2
  (0x00007fc61661c000)
  libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fc616338000)
  libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fc616134000)
  libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fc615f02000)
  libdl.so.2 => /lib64/libdl.so.2 (0x00007fc615cfd000)
  libz.so.1 => /lib64/libz.so.1 (0x00007fc615ae7000)
  /lib64/ld-linux-x86-64.so.2 (0x00007fc6174a0000)
  libkrb5support.so.0 => /lib64/libkrb5support.so.0
  (0x00007fc6158d8000)
  libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fc6156d3000)
  libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fc6154b9000)
  libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc61529d000)
  libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fc615077000)
  libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fc614e16000)
  liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fc614bf1000)

  Client:

  [root@ops:~] #ldd /usr/local/nagios/libexec/check_nrpe
  * libssl.so.6 => /lib64/libssl.so.6 (0x00002aaaaaaba000)*
  * libcrypto.so.6 => /lib64/libcrypto.so.6 (0x00002aaaaad08000)*
  libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaab05a000)
  libc.so.6 => /lib64/libc.so.6 (0x00002aaaab273000)
  libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2
  (0x00002aaaab5cc000)
  libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00002aaaab7fa000)
  libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00002aaaaba90000)
  libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00002aaaabc92000)
  libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaabeb7000)
  libz.so.1 => /lib64/libz.so.1 (0x00002aaaac0bc000)
  /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
  libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x0
  0002aaaac2d0000)
  libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00002aaaac4d8000)
  libresolv.so.2 => /lib64/libresolv.so.2 (0x00002aaaac6db000)
  libselinux.so.1 => /lib64/libselinux.so.1 (0x00002aaaac8f0000)
  libsepol.so.1 => /lib64/libsepol.so.1 (0x00002aaaacb09000)

  So it looks like everything is OK from the SSL end of things. Any other ideas or suggestions?

  Thanks Tim

• well. how about trying default setting and running nrped without xinetd.

• Oh my mistake. I mean nrpe without parameters. It should say something about SSL/TLS aktiv or so. You could test nrpe without SSL. Use nrpe -n – H host Am 01.05.2015 13:18 schrieb “Eero Volotinen” :

• Does /usr/local/nagios/etc/nrpe.cfg exist and is it readable by user or group ‘nagios’? Did the user:group ‘nagios’ get created when you did the installation? Those were my two routine stumbles before I automated rollouts.

  Regards,

• Yes, also it could be nagios use another configs location. Check: whereis nagios. Am 01.05.2015 13:44 schrieb “Brian Miller” :

• It sounds like you’ve got NRPE up on your AWS system, so I think you might need to take a closer look at your security groups to make sure it is allowing the NRPE port in from the source you’re checking from. You could always check with a check_nrpe from another host in the same VPC if you want to make sure its not NRPE configuration-related.

• This is what I see about ssl if I just run nrpe on the client without any flags:

  [root@ops:~] #nrpe| head -8

  NRPE – Nagios Remote Plugin Executor Copyright (c) 1999-2008 Ethan Galstad (nagios@nagios.org)
  Version: 2.15
  Last Modified: 09-06-2013
  License: GPL v2 with exemptions (-l for more info)
  SSL/TLS Available: Anonymous DH Mode, OpenSSL 0.9.6 or higher required TCP Wrappers Available

  And if I go back to the monitoring host and try to run nrpe with the -n flag, this is what I get:

  [root@monitor1:~] #/usr/local/nagios/libexec/check_nrpe -n -H
  ops.jokefire.com
  *CHECK_NRPE: Error receiving data from daemon.*

  And still getting the SSL error without the -n flag:

  [root@monitor1:~] #/usr/local/nagios/libexec/check_nrpe -H ops.jokefire.com
  *CHECK_NRPE: Error – Could not complete SSL handshake.*

  Running nmap from the monitor host I can see that the nrpe port is open:

  [root@monitor1:~] #nmap -p 5666 ops.jokefire.com

  Starting Nmap 6.40 ( http://nmap.org ) at 2015-05-01 12:38 EDT
  Nmap scan report for ops.jokefire.com (54.225.218.125)
  Host is up (0.011s latency). rDNS record for 54.225.218.125: ec2-54-225-218-125.compute-1.amazonaws.com PORT STATE SERVICE
  *5666/tcp open nrpe*

  Nmap done: 1 IP address (1 host up) scanned in 0.14 seconds

  Yet if I try telnetting to it, it connects, then closes the connection immediately:

  [root@monitor1:~] #telnet ops.jokefire.com 5666
  Trying 54.225.218.125…
  *Connected to ops.jokefire.com <http://ops.jokefire.com>.*
  Escape character is ‘^]’.
  *Connection closed by foreign host.*

  Going back to the ops host that I want to monitor, I can verify that the port is listening:

  [root@ops:~] #lsof -i :5666
  COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
  xinetd 1434 root 5u IPv4 4063 TCP *:nrpe (LISTEN)

  And I can verify that the nrpe conf is owned by the nagios user and group:

  [root@ops:~] #ls -l /usr/local/nagios/etc/nrpe.cfg
  -rw-r–r– 1 nagios nagios 7988 May 1 00:37 /usr/local/nagios/etc/nrpe.cfg

  I think that covers all your suggestions. Except for Eero’s suggestion to try running nrpe without xinetd. I can try to get to that later, but I may not have time for that suggestion today. But as I demonstrate above, the problem is not that nrpe isn’t listening.

  This remains a really odd situation. Does anyone else have any clues?

  Thanks, Tim

• Hi

  NRPE: Error receiving data from daemon

  Seems as this is not a SSL Problem. Do you have a nagios user account? Cat
  /etc/passwd Am 01.05.2015 18:45 schrieb “Tim Dunphy” :

• Hi Eric,

  Yep! Both hosts have nagios user accounts.

  Demonstrating from the client:

  [root@ops:~] #id nagios uid 02(nagios) gid 02(nagios) groups 02(nagios),2008(nagioscmd)

  And this is from the monitoring server:

  [root@monitor1:~] #id nagios uid01(nagios) gid01(nagios) groups01(nagios),1002(nagcmd)

  I do notice a slight difference in the user id and group id numbers. But I
  don’t think that could be causing any issue. Does anyone else disagree?

  I might want to standardize user accounts at some point howver.

  Thanks!
  Tim

• Is it working on localhost with nrpe check? Did you checked out logs of nrped?

  Eero
  1.5.2015 8.31 ip. “Tim Dunphy” kirjoitti:

• Does “iptables -L” show anything of note?

• Hi Brian,

  Does “iptables -L” show anything of note?

  I’m leaving iptables off in this host. Because it’s an AWS EC2 host I’m managing the firewall ports using the AWS security groups.

  [root@ops:~] #service iptables status Firewall is stopped.

  But still, there’s this…

  [root@monitor1:~] #/usr/local/nagios/libexec/check_nrpe -H ops.jokefire.com CHECK_NRPE: Error – Could not complete SSL handshake.

  Sadly…. :(

  Thanks for your input tho!

• Does ‘ldd /usr/local/nagios/bin/nrpe’ show any missing libs?

• Hi Brian,

  Does ‘ldd /usr/local/nagios/bin/nrpe’ show any missing libs?

  Well, the NRPE binary looks good both on the client and the server from what I can tell:

  Client:

  [root@ops:~] #ldd /usr/local/nagios/bin/nrpe
  libssl.so.6 => /lib64/libssl.so.6 (0x00002aaaaaaba000)
  libcrypto.so.6 => /lib64/libcrypto.so.6 (0x00002aaaaad08000)
  libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaab05a000)
  libwrap.so.0 => /lib64/libwrap.so.0 (0x00002aaaab273000)
  libc.so.6 => /lib64/libc.so.6 (0x00002aaaab47c000)
  libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2
  (0x00002aaaab7d5000)
  libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00002aaaaba04000)
  libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00002aaaabc99000)
  libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00002aaaabe9b000)
  libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaac0c1000)
  libz.so.1 => /lib64/libz.so.1 (0x00002aaaac2c5000)
  /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
  libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0
  (0x00002aaaac4d9000)
  libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00002aaaac6e2000)
  libresolv.so.2 => /lib64/libresolv.so.2 (0x00002aaaac8e4000)
  libselinux.so.1 => /lib64/libselinux.so.1 (0x00002aaaacafa000)
  libsepol.so.1 => /lib64/libsepol.so.1 (0x00002aaaacd12000)

  And server:

  [root@monitor1:~] #ldd /usr/local/nagios/bin/nrpe
  linux-vdso.so.1 => (0x00007fffffffd000)
  libssl.so.10 => /lib64/libssl.so.10 (0x00007fdd51590000)
  libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fdd511a9000)
  libnsl.so.1 => /lib64/libnsl.so.1 (0x00007fdd50f8f000)
  libc.so.6 => /lib64/libc.so.6 (0x00007fdd50bce000)
  libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2
  (0x00007fdd50982000)
  libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fdd5069e000)
  libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fdd5049a000)
  libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fdd50268000)
  libdl.so.2 => /lib64/libdl.so.2 (0x00007fdd50063000)
  libz.so.1 => /lib64/libz.so.1 (0x00007fdd4fe4d000)
  /lib64/ld-linux-x86-64.so.2 (0x00007fdd51806000)
  libkrb5support.so.0 => /lib64/libkrb5support.so.0
  (0x00007fdd4fc3e000)
  libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fdd4fa39000)
  libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fdd4f81f000)
  libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdd4f603000)
  libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fdd4f3dd000)
  libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fdd4f17c000)
  liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fdd4ef57000)

  Both look completely fine! No missing libs. But thanks for the suggestion tho! Definitely not a bad idea to rule that out!

  Thanks, Tim

• Hmmm… you make an excellent point! I picked up this habit from an AWS
  shop I used to work at. But what you just said will make me reconsider!

  I tailed /var/log/messages while hitting the client with check_nrpe from the monitoring host. However, that didn’t cause an entry in the messages log.

  Yep! I’ve set the only_from to have only the loopback address and the IP
  for the monitoring host in /etc/xinetd.d/npre.

  Cool! Thanks. I’ll check it out, and see if I can find anything useful.

  I appreciate the input!

  Also I really appreciate the ongoing dialog with the community on this issue. I’m grasping at straws at this point. And all the attempts at help have been really great! I hope we can still get to the bottom of this!

  Tim

• Hmmm I don’t find any log specific to nrpe. In other words I don’t see
  /var/log/nrpe.log or whatever. :)

  And when I tail -f /var/log/secure or /var/log/messages I don’t see any entries turning up in them when I hit the client with check_nrpe. I was checking the logs on the client itself.

  Hmmmm. but the nrpe.confg file is ignored in the case of allowed hosts. From the nrpe config:

  # NOTE: This option is ignored if NRPE is running under either inetd or xinetd

  allowed_hosts7.0.0.1

  Thanks for the input tho, I genuinely appreciate it!

• Are xinetd log entries written when you connect from localhost?

• Hi

  Some more points:
  Does the user nagios have rights to nrpe binary and config file?
  Check nrpe.cfg for nrpe_user=nagios and nrpe_group=nagios.

  To activate logging:
  As default, nrpe log to syslog BUT you have to add daemon.debug to
  /etc/rsyslog.conf :
  **.info;mail.none;authpriv.none;cron.none;daemon.debug
  /var/log/messages*
  And set debug=1 in nrpe.cfg service rsyslog restart service nrpe restart

  Regards, Eric

  2015-05-03 6:37 GMT+02:00 Jonathan Billings :

• Don’t if these links are of any help but worth checking (if you have not done so)
  <http://assets.nagios.com/downloads/nagiosxi/docs/NRPE-Troubleshooting-and-Common-Solutions.pdf>
  and
  <http://geekpeek.net/could-not-complete-ssl-handshake/>

  — Arun Khan