Could Not Complete SSL Handshake To Amazon EC2 Host

Home » CentOS » Could Not Complete SSL Handshake To Amazon EC2 Host
CentOS 22 Comments

Hello,

I am trying to monitor a host in the Amazon EC2 cloud.

Yet when I try to check NRPE from the monitoring host I am getting an SSL
handshake error:

[root@monitor1:~] #/usr/local/nagios/libexec/check_nrpe -H ops.jokefire.com CHECK_NRPE: Error – Could not complete SSL handshake.

And if I telnet into the host on port 5666 to see if the FW port is open, the connection closes right away:

[root@monitor1:~] #telnet ops.somewhere.com 5666
Trying 54.225.218.125… Connected to ops.somewhere.com. Escape character is ‘^]’. Connection closed by foreign host.

You can see there it connects, but then it closes immediately after the connection.

I have NRPE running on the host I want to monitor:

[root@ops:~] #lsof -i :5666
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
xinetd 1434 root 5u IPv4 4063 TCP *:nrpe (LISTEN)

And I have the IP of my nagios server listed in the xinetd conf file:

[root@ops:~] #cat /etc/xinetd.d/nrpe
# default: on
# description: NRPE (Nagios Remote Plugin Executor)
service nrpe
{
flags = REUSE
socket_type = stream
port = 5666
wait = no
user = nagios
group = nagios
server = /usr/local/nagios/bin/nrpe
server_args = -c /usr/local/nagios/etc/nrpe.cfg –inetd
log_on_failure += USERID
disable = no
only_from = 127.0.0.1 xx.xx.xx.xx # <- representing my real nagios server IP } And I have my default security group for that host open on port 5666 to the world for this experiment. I plan on locking that down again to the single IP of my monitoring host once I get this resolved. Does anyone have any suggestions on how I can get that problem solved? Thanks, Tim

22 thoughts on - Could Not Complete SSL Handshake To Amazon EC2 Host

  • Hi Does the deamon run under xinetd? Then you have to configure the only_from in */etc/**xinetd.d**/**nrpe* to.

    Regards Eric Am 01.05.2015 06:46 schrieb “Tim Dunphy” :

  • Hi Eric,

    Thanks for your reply. I do have nrpe running under xinetd on the host I’m trying to monitor.

    And running the nrpe checl locally:

    [root@ops:~] #/usr/local/nagios/libexec/check_nrpe -H localhost NRPE v2.15

    [root@ops:~] #grep only_from /etc/xinetd.d/nrpe
    only_from = 127.0.0.1 216.120.248.126

    And I do have port 5666 open on the security group for this host.

    And I made sure the local firewall was stopped, because I am blocking ports with the security groups instead.

    [root@ops:~] #service iptables status Firewall is stopped.

    It’s only when checking from the monitoring host that nrpe fails:

    [root@monitor1:~] #/usr/local/nagios/libexec/check_nrpe -H ops.jokefire.com CHECK_NRPE: Error – Could not complete SSL handshake.

    Really, really puzzling. This is driving me up a wall!! I hopeI can solve this soon….

    Thanks for any and all help with this one!!
    Tim

  • This is strange… Do you have SSL aktive on both systems? Run nrpr localy without parameters
    (this should return some nrpe stats) and check ldd for libssl. Am 01.05.2015 07:32 schrieb “Tim Dunphy” :

  • I don’t seem to have that command.

    [root@monitor1:~] #find / -name “*nrpr” 2> /dev/null
    [root@monitor1:~] #

    And that’s on either system.

    And if I do an ldd on both, this is what I can tell:

    Server:

    [root@monitor1:~] #ldd /usr/local/nagios/libexec/check_nrpe
    linux-vdso.so.1 => (0x00007fffd895d000)
    * libssl.so.10 => /lib64/libssl.so.10 (0x00007fc61722a000)*
    * libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fc616e43000)*
    libnsl.so.1 => /lib64/libnsl.so.1 (0x00007fc616c29000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fc616868000)
    libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2
    (0x00007fc61661c000)
    libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fc616338000)
    libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fc616134000)
    libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fc615f02000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fc615cfd000)
    libz.so.1 => /lib64/libz.so.1 (0x00007fc615ae7000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fc6174a0000)
    libkrb5support.so.0 => /lib64/libkrb5support.so.0
    (0x00007fc6158d8000)
    libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fc6156d3000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fc6154b9000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc61529d000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fc615077000)
    libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fc614e16000)
    liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fc614bf1000)

    Client:

    [root@ops:~] #ldd /usr/local/nagios/libexec/check_nrpe
    * libssl.so.6 => /lib64/libssl.so.6 (0x00002aaaaaaba000)*
    * libcrypto.so.6 => /lib64/libcrypto.so.6 (0x00002aaaaad08000)*
    libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaab05a000)
    libc.so.6 => /lib64/libc.so.6 (0x00002aaaab273000)
    libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2
    (0x00002aaaab5cc000)
    libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00002aaaab7fa000)
    libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00002aaaaba90000)
    libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00002aaaabc92000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaabeb7000)
    libz.so.1 => /lib64/libz.so.1 (0x00002aaaac0bc000)
    /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
    libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x0
    0002aaaac2d0000)
    libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00002aaaac4d8000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00002aaaac6db000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00002aaaac8f0000)
    libsepol.so.1 => /lib64/libsepol.so.1 (0x00002aaaacb09000)

    So it looks like everything is OK from the SSL end of things. Any other ideas or suggestions?

    Thanks Tim

  • Oh my mistake. I mean nrpe without parameters. It should say something about SSL/TLS aktiv or so. You could test nrpe without SSL. Use nrpe -n – H host Am 01.05.2015 13:18 schrieb “Eero Volotinen” :

  • Does /usr/local/nagios/etc/nrpe.cfg exist and is it readable by user or group ‘nagios’? Did the user:group ‘nagios’ get created when you did the installation? Those were my two routine stumbles before I automated rollouts.

    Regards,

  • Yes, also it could be nagios use another configs location. Check: whereis nagios. Am 01.05.2015 13:44 schrieb “Brian Miller” :

  • It sounds like you’ve got NRPE up on your AWS system, so I think you might need to take a closer look at your security groups to make sure it is allowing the NRPE port in from the source you’re checking from. You could always check with a check_nrpe from another host in the same VPC if you want to make sure its not NRPE configuration-related.

  • This is what I see about ssl if I just run nrpe on the client without any flags:

    [root@ops:~] #nrpe| head -8

    NRPE – Nagios Remote Plugin Executor Copyright (c) 1999-2008 Ethan Galstad (nagios@nagios.org)
    Version: 2.15
    Last Modified: 09-06-2013
    License: GPL v2 with exemptions (-l for more info)
    SSL/TLS Available: Anonymous DH Mode, OpenSSL 0.9.6 or higher required TCP Wrappers Available

    And if I go back to the monitoring host and try to run nrpe with the -n flag, this is what I get:

    [root@monitor1:~] #/usr/local/nagios/libexec/check_nrpe -n -H
    ops.jokefire.com
    *CHECK_NRPE: Error receiving data from daemon.*

    And still getting the SSL error without the -n flag:

    [root@monitor1:~] #/usr/local/nagios/libexec/check_nrpe -H ops.jokefire.com
    *CHECK_NRPE: Error – Could not complete SSL handshake.*

    Running nmap from the monitor host I can see that the nrpe port is open:

    [root@monitor1:~] #nmap -p 5666 ops.jokefire.com

    Starting Nmap 6.40 ( http://nmap.org ) at 2015-05-01 12:38 EDT
    Nmap scan report for ops.jokefire.com (54.225.218.125)
    Host is up (0.011s latency). rDNS record for 54.225.218.125: ec2-54-225-218-125.compute-1.amazonaws.com PORT STATE SERVICE
    *5666/tcp open nrpe*

    Nmap done: 1 IP address (1 host up) scanned in 0.14 seconds

    Yet if I try telnetting to it, it connects, then closes the connection immediately:

    [root@monitor1:~] #telnet ops.jokefire.com 5666
    Trying 54.225.218.125…
    *Connected to ops.jokefire.com < http://ops.jokefire.com>.*
    Escape character is ‘^]’.
    *Connection closed by foreign host.*

    Going back to the ops host that I want to monitor, I can verify that the port is listening:

    [root@ops:~] #lsof -i :5666
    COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
    xinetd 1434 root 5u IPv4 4063 TCP *:nrpe (LISTEN)

    And I can verify that the nrpe conf is owned by the nagios user and group:

    [root@ops:~] #ls -l /usr/local/nagios/etc/nrpe.cfg
    -rw-r–r– 1 nagios nagios 7988 May 1 00:37 /usr/local/nagios/etc/nrpe.cfg

    I think that covers all your suggestions. Except for Eero’s suggestion to try running nrpe without xinetd. I can try to get to that later, but I may not have time for that suggestion today. But as I demonstrate above, the problem is not that nrpe isn’t listening.

    This remains a really odd situation. Does anyone else have any clues?

    Thanks, Tim

  • Hi

    NRPE: Error receiving data from daemon

    Seems as this is not a SSL Problem. Do you have a nagios user account? Cat
    /etc/passwd Am 01.05.2015 18:45 schrieb “Tim Dunphy” :

  • Hi Eric,

    Yep! Both hosts have nagios user accounts.

    Demonstrating from the client:

    [root@ops:~] #id nagios uid 02(nagios) gid 02(nagios) groups 02(nagios),2008(nagioscmd)

    And this is from the monitoring server:

    [root@monitor1:~] #id nagios uid01(nagios) gid01(nagios) groups01(nagios),1002(nagcmd)

    I do notice a slight difference in the user id and group id numbers. But I
    don’t think that could be causing any issue. Does anyone else disagree?

    I might want to standardize user accounts at some point howver.

    Thanks!
    Tim

  • Is it working on localhost with nrpe check? Did you checked out logs of nrped?

    Eero
    1.5.2015 8.31 ip. “Tim Dunphy” kirjoitti:

  • Hi Brian,

    Does “iptables -L” show anything of note?

    I’m leaving iptables off in this host. Because it’s an AWS EC2 host I’m managing the firewall ports using the AWS security groups.

    [root@ops:~] #service iptables status Firewall is stopped.

    But still, there’s this…

    [root@monitor1:~] #/usr/local/nagios/libexec/check_nrpe -H ops.jokefire.com CHECK_NRPE: Error – Could not complete SSL handshake.

    Sadly…. :(

    Thanks for your input tho!

  • Hi Brian,

    Does ‘ldd /usr/local/nagios/bin/nrpe’ show any missing libs?

    Well, the NRPE binary looks good both on the client and the server from what I can tell:

    Client:

    [root@ops:~] #ldd /usr/local/nagios/bin/nrpe
    libssl.so.6 => /lib64/libssl.so.6 (0x00002aaaaaaba000)
    libcrypto.so.6 => /lib64/libcrypto.so.6 (0x00002aaaaad08000)
    libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaab05a000)
    libwrap.so.0 => /lib64/libwrap.so.0 (0x00002aaaab273000)
    libc.so.6 => /lib64/libc.so.6 (0x00002aaaab47c000)
    libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2
    (0x00002aaaab7d5000)
    libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00002aaaaba04000)
    libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00002aaaabc99000)
    libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00002aaaabe9b000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaac0c1000)
    libz.so.1 => /lib64/libz.so.1 (0x00002aaaac2c5000)
    /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
    libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0
    (0x00002aaaac4d9000)
    libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00002aaaac6e2000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00002aaaac8e4000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00002aaaacafa000)
    libsepol.so.1 => /lib64/libsepol.so.1 (0x00002aaaacd12000)

    And server:

    [root@monitor1:~] #ldd /usr/local/nagios/bin/nrpe
    linux-vdso.so.1 => (0x00007fffffffd000)
    libssl.so.10 => /lib64/libssl.so.10 (0x00007fdd51590000)
    libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fdd511a9000)
    libnsl.so.1 => /lib64/libnsl.so.1 (0x00007fdd50f8f000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fdd50bce000)
    libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2
    (0x00007fdd50982000)
    libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fdd5069e000)
    libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fdd5049a000)
    libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fdd50268000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fdd50063000)
    libz.so.1 => /lib64/libz.so.1 (0x00007fdd4fe4d000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fdd51806000)
    libkrb5support.so.0 => /lib64/libkrb5support.so.0
    (0x00007fdd4fc3e000)
    libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fdd4fa39000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fdd4f81f000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdd4f603000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fdd4f3dd000)
    libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fdd4f17c000)
    liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fdd4ef57000)

    Both look completely fine! No missing libs. But thanks for the suggestion tho! Definitely not a bad idea to rule that out!

    Thanks, Tim

  • Hmmm… you make an excellent point! I picked up this habit from an AWS
    shop I used to work at. But what you just said will make me reconsider!

    I tailed /var/log/messages while hitting the client with check_nrpe from the monitoring host. However, that didn’t cause an entry in the messages log.

    Yep! I’ve set the only_from to have only the loopback address and the IP
    for the monitoring host in /etc/xinetd.d/npre.

    Cool! Thanks. I’ll check it out, and see if I can find anything useful.

    I appreciate the input!

    Also I really appreciate the ongoing dialog with the community on this issue. I’m grasping at straws at this point. And all the attempts at help have been really great! I hope we can still get to the bottom of this!

    Tim

  • Hmmm I don’t find any log specific to nrpe. In other words I don’t see
    /var/log/nrpe.log or whatever. :)

    And when I tail -f /var/log/secure or /var/log/messages I don’t see any entries turning up in them when I hit the client with check_nrpe. I was checking the logs on the client itself.

    Hmmmm. but the nrpe.confg file is ignored in the case of allowed hosts. From the nrpe config:

    # NOTE: This option is ignored if NRPE is running under either inetd or xinetd

    allowed_hosts7.0.0.1

    Thanks for the input tho, I genuinely appreciate it!

  • Hi

    Some more points:
    Does the user nagios have rights to nrpe binary and config file?
    Check nrpe.cfg for nrpe_user=nagios and nrpe_group=nagios.

    To activate logging:
    As default, nrpe log to syslog BUT you have to add daemon.debug to
    /etc/rsyslog.conf :
    **.info;mail.none;authpriv.none;cron.none;daemon.debug
    /var/log/messages*
    And set debug=1 in nrpe.cfg service rsyslog restart service nrpe restart

    Regards, Eric

    2015-05-03 6:37 GMT+02:00 Jonathan Billings :

LEAVE A COMMENT