Could Not Complete SSL Handshake To Amazon EC2 Host
Hello,
I am trying to monitor a host in the Amazon EC2 cloud.
Yet when I try to check NRPE from the monitoring host I am getting an SSL
handshake error:
[root@monitor1:~] #/usr/local/nagios/libexec/check_nrpe -H ops.jokefire.com CHECK_NRPE: Error – Could not complete SSL handshake.
And if I telnet into the host on port 5666 to see if the FW port is open, the connection closes right away:
[root@monitor1:~] #telnet ops.somewhere.com 5666
Trying 54.225.218.125… Connected to ops.somewhere.com. Escape character is ‘^]’. Connection closed by foreign host.
You can see there it connects, but then it closes immediately after the connection.
I have NRPE running on the host I want to monitor:
[root@ops:~] #lsof -i :5666
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
xinetd 1434 root 5u IPv4 4063 TCP *:nrpe (LISTEN)
And I have the IP of my nagios server listed in the xinetd conf file:
[root@ops:~] #cat /etc/xinetd.d/nrpe
# default: on
# description: NRPE (Nagios Remote Plugin Executor)
service nrpe
{
flags = REUSE
socket_type = stream
port = 5666
wait = no
user = nagios
group = nagios
server = /usr/local/nagios/bin/nrpe
server_args = -c /usr/local/nagios/etc/nrpe.cfg –inetd
log_on_failure += USERID
disable = no
only_from = 127.0.0.1 xx.xx.xx.xx # <- representing my real nagios server IP
}
And I have my default security group for that host open on port 5666 to the world for this experiment. I plan on locking that down again to the single IP of my monitoring host once I get this resolved.
Does anyone have any suggestions on how I can get that problem solved?
Thanks, Tim
22 thoughts on - Could Not Complete SSL Handshake To Amazon EC2 Host
Hi Does the deamon run under xinetd? Then you have to configure the only_from in */etc/**xinetd.d**/**nrpe* to.
Regards Eric Am 01.05.2015 06:46 schrieb “Tim Dunphy”:
Hi Eric,
Thanks for your reply. I do have nrpe running under xinetd on the host I’m trying to monitor.
And running the nrpe checl locally:
[root@ops:~] #/usr/local/nagios/libexec/check_nrpe -H localhost NRPE v2.15
[root@ops:~] #grep only_from /etc/xinetd.d/nrpe
only_from = 127.0.0.1 216.120.248.126
And I do have port 5666 open on the security group for this host.
And I made sure the local firewall was stopped, because I am blocking ports with the security groups instead.
[root@ops:~] #service iptables status Firewall is stopped.
It’s only when checking from the monitoring host that nrpe fails:
[root@monitor1:~] #/usr/local/nagios/libexec/check_nrpe -H ops.jokefire.com CHECK_NRPE: Error – Could not complete SSL handshake.
Really, really puzzling. This is driving me up a wall!! I hopeI can solve this soon….
Thanks for any and all help with this one!!
Tim
This is strange… Do you have SSL aktive on both systems? Run nrpr localy without parameters:
(this should return some nrpe stats) and check ldd for libssl. Am 01.05.2015 07:32 schrieb “Tim Dunphy”
I don’t seem to have that command.
[root@monitor1:~] #find / -name “*nrpr” 2> /dev/null
[root@monitor1:~] #
And that’s on either system.
And if I do an ldd on both, this is what I can tell:
Server:
[root@monitor1:~] #ldd /usr/local/nagios/libexec/check_nrpe
linux-vdso.so.1 => (0x00007fffd895d000)
* libssl.so.10 => /lib64/libssl.so.10 (0x00007fc61722a000)*
* libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fc616e43000)*
libnsl.so.1 => /lib64/libnsl.so.1 (0x00007fc616c29000)
libc.so.6 => /lib64/libc.so.6 (0x00007fc616868000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2
(0x00007fc61661c000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fc616338000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fc616134000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fc615f02000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fc615cfd000)
libz.so.1 => /lib64/libz.so.1 (0x00007fc615ae7000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc6174a0000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0
(0x00007fc6158d8000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fc6156d3000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fc6154b9000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fc61529d000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fc615077000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fc614e16000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fc614bf1000)
Client:
[root@ops:~] #ldd /usr/local/nagios/libexec/check_nrpe
* libssl.so.6 => /lib64/libssl.so.6 (0x00002aaaaaaba000)*
* libcrypto.so.6 => /lib64/libcrypto.so.6 (0x00002aaaaad08000)*
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaab05a000)
libc.so.6 => /lib64/libc.so.6 (0x00002aaaab273000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2
(0x00002aaaab5cc000)
libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00002aaaab7fa000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00002aaaaba90000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00002aaaabc92000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaabeb7000)
libz.so.1 => /lib64/libz.so.1 (0x00002aaaac0bc000)
/lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x0
0002aaaac2d0000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00002aaaac4d8000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00002aaaac6db000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00002aaaac8f0000)
libsepol.so.1 => /lib64/libsepol.so.1 (0x00002aaaacb09000)
So it looks like everything is OK from the SSL end of things. Any other ideas or suggestions?
Thanks Tim
well. how about trying default setting and running nrped without xinetd.
Oh my mistake. I mean nrpe without parameters. It should say something about SSL/TLS aktiv or so. You could test nrpe without SSL. Use nrpe -n – H host Am 01.05.2015 13:18 schrieb “Eero Volotinen”:
Does /usr/local/nagios/etc/nrpe.cfg exist and is it readable by user or group ‘nagios’? Did the user:group ‘nagios’ get created when you did the installation? Those were my two routine stumbles before I automated rollouts.
Regards,
Yes, also it could be nagios use another configs location. Check: whereis nagios. Am 01.05.2015 13:44 schrieb “Brian Miller”:
It sounds like you’ve got NRPE up on your AWS system, so I think you might need to take a closer look at your security groups to make sure it is allowing the NRPE port in from the source you’re checking from. You could always check with a check_nrpe from another host in the same VPC if you want to make sure its not NRPE configuration-related.
This is what I see about ssl if I just run nrpe on the client without any flags:
[root@ops:~] #nrpe| head -8
NRPE – Nagios Remote Plugin Executor Copyright (c) 1999-2008 Ethan Galstad (nagios@nagios.org)
Version: 2.15
Last Modified: 09-06-2013
License: GPL v2 with exemptions (-l for more info)
SSL/TLS Available: Anonymous DH Mode, OpenSSL 0.9.6 or higher required TCP Wrappers Available
And if I go back to the monitoring host and try to run nrpe with the -n flag, this is what I get:
[root@monitor1:~] #/usr/local/nagios/libexec/check_nrpe -n -H
ops.jokefire.com
*CHECK_NRPE: Error receiving data from daemon.*
And still getting the SSL error without the -n flag:
[root@monitor1:~] #/usr/local/nagios/libexec/check_nrpe -H ops.jokefire.com
*CHECK_NRPE: Error – Could not complete SSL handshake.*
Running nmap from the monitor host I can see that the nrpe port is open:
[root@monitor1:~] #nmap -p 5666 ops.jokefire.com
Starting Nmap 6.40 ( http://nmap.org ) at 2015-05-01 12:38 EDT
Nmap scan report for ops.jokefire.com (54.225.218.125)
Host is up (0.011s latency). rDNS record for 54.225.218.125: ec2-54-225-218-125.compute-1.amazonaws.com PORT STATE SERVICE
*5666/tcp open nrpe*
Nmap done: 1 IP address (1 host up) scanned in 0.14 seconds
Yet if I try telnetting to it, it connects, then closes the connection immediately:
[root@monitor1:~] #telnet ops.jokefire.com 5666
Trying 54.225.218.125…
*Connected to ops.jokefire.com <http://ops.jokefire.com>.*
Escape character is ‘^]’.
*Connection closed by foreign host.*
Going back to the ops host that I want to monitor, I can verify that the port is listening:
[root@ops:~] #lsof -i :5666
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
xinetd 1434 root 5u IPv4 4063 TCP *:nrpe (LISTEN)
And I can verify that the nrpe conf is owned by the nagios user and group:
[root@ops:~] #ls -l /usr/local/nagios/etc/nrpe.cfg
-rw-r–r– 1 nagios nagios 7988 May 1 00:37 /usr/local/nagios/etc/nrpe.cfg
I think that covers all your suggestions. Except for Eero’s suggestion to try running nrpe without xinetd. I can try to get to that later, but I may not have time for that suggestion today. But as I demonstrate above, the problem is not that nrpe isn’t listening.
This remains a really odd situation. Does anyone else have any clues?
Thanks, Tim
Hi
NRPE: Error receiving data from daemon
Seems as this is not a SSL Problem. Do you have a nagios user account? Cat:
/etc/passwd Am 01.05.2015 18:45 schrieb “Tim Dunphy”
Hi Eric,
Yep! Both hosts have nagios user accounts.
Demonstrating from the client:
[root@ops:~] #id nagios uid 02(nagios) gid 02(nagios) groups 02(nagios),2008(nagioscmd)
And this is from the monitoring server:
[root@monitor1:~] #id nagios uid01(nagios) gid01(nagios) groups01(nagios),1002(nagcmd)
I do notice a slight difference in the user id and group id numbers. But I
don’t think that could be causing any issue. Does anyone else disagree?
I might want to standardize user accounts at some point howver.
Thanks!
Tim
Is it working on localhost with nrpe check? Did you checked out logs of nrped?
Eero kirjoitti:
1.5.2015 8.31 ip. “Tim Dunphy”
Does “iptables -L” show anything of note?
Hi Brian,
Does “iptables -L” show anything of note?
I’m leaving iptables off in this host. Because it’s an AWS EC2 host I’m managing the firewall ports using the AWS security groups.
[root@ops:~] #service iptables status Firewall is stopped.
But still, there’s this…
[root@monitor1:~] #/usr/local/nagios/libexec/check_nrpe -H ops.jokefire.com CHECK_NRPE: Error – Could not complete SSL handshake.
Sadly…. :(
Thanks for your input tho!
Does ‘ldd /usr/local/nagios/bin/nrpe’ show any missing libs?
Hi Brian,
Does ‘ldd /usr/local/nagios/bin/nrpe’ show any missing libs?
Well, the NRPE binary looks good both on the client and the server from what I can tell:
Client:
[root@ops:~] #ldd /usr/local/nagios/bin/nrpe
libssl.so.6 => /lib64/libssl.so.6 (0x00002aaaaaaba000)
libcrypto.so.6 => /lib64/libcrypto.so.6 (0x00002aaaaad08000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00002aaaab05a000)
libwrap.so.0 => /lib64/libwrap.so.0 (0x00002aaaab273000)
libc.so.6 => /lib64/libc.so.6 (0x00002aaaab47c000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2
(0x00002aaaab7d5000)
libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00002aaaaba04000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00002aaaabc99000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00002aaaabe9b000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002aaaac0c1000)
libz.so.1 => /lib64/libz.so.1 (0x00002aaaac2c5000)
/lib64/ld-linux-x86-64.so.2 (0x0000555555554000)
libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0
(0x00002aaaac4d9000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00002aaaac6e2000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00002aaaac8e4000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00002aaaacafa000)
libsepol.so.1 => /lib64/libsepol.so.1 (0x00002aaaacd12000)
And server:
[root@monitor1:~] #ldd /usr/local/nagios/bin/nrpe
linux-vdso.so.1 => (0x00007fffffffd000)
libssl.so.10 => /lib64/libssl.so.10 (0x00007fdd51590000)
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007fdd511a9000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x00007fdd50f8f000)
libc.so.6 => /lib64/libc.so.6 (0x00007fdd50bce000)
libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2
(0x00007fdd50982000)
libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007fdd5069e000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fdd5049a000)
libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007fdd50268000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fdd50063000)
libz.so.1 => /lib64/libz.so.1 (0x00007fdd4fe4d000)
/lib64/ld-linux-x86-64.so.2 (0x00007fdd51806000)
libkrb5support.so.0 => /lib64/libkrb5support.so.0
(0x00007fdd4fc3e000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007fdd4fa39000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fdd4f81f000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fdd4f603000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fdd4f3dd000)
libpcre.so.1 => /lib64/libpcre.so.1 (0x00007fdd4f17c000)
liblzma.so.5 => /lib64/liblzma.so.5 (0x00007fdd4ef57000)
Both look completely fine! No missing libs. But thanks for the suggestion tho! Definitely not a bad idea to rule that out!
Thanks, Tim
Hmmm… you make an excellent point! I picked up this habit from an AWS
shop I used to work at. But what you just said will make me reconsider!
I tailed /var/log/messages while hitting the client with check_nrpe from the monitoring host. However, that didn’t cause an entry in the messages log.
Yep! I’ve set the only_from to have only the loopback address and the IP
for the monitoring host in /etc/xinetd.d/npre.
Cool! Thanks. I’ll check it out, and see if I can find anything useful.
I appreciate the input!
Also I really appreciate the ongoing dialog with the community on this issue. I’m grasping at straws at this point. And all the attempts at help have been really great! I hope we can still get to the bottom of this!
Tim
Hmmm I don’t find any log specific to nrpe. In other words I don’t see
/var/log/nrpe.log or whatever. :)
And when I tail -f /var/log/secure or /var/log/messages I don’t see any entries turning up in them when I hit the client with check_nrpe. I was checking the logs on the client itself.
Hmmmm. but the nrpe.confg file is ignored in the case of allowed hosts. From the nrpe config:
# NOTE: This option is ignored if NRPE is running under either inetd or xinetd
allowed_hosts7.0.0.1
Thanks for the input tho, I genuinely appreciate it!
Are xinetd log entries written when you connect from localhost?
Hi
Some more points:
Does the user nagios have rights to nrpe binary and config file?
Check nrpe.cfg for nrpe_user=nagios and nrpe_group=nagios.
To activate logging:
As default, nrpe log to syslog BUT you have to add daemon.debug to
/etc/rsyslog.conf :
**.info;mail.none;authpriv.none;cron.none;daemon.debug
/var/log/messages*
And set debug=1 in nrpe.cfg service rsyslog restart service nrpe restart
Regards, Eric
2015-05-03 6:37 GMT+02:00 Jonathan Billings:
Don’t if these links are of any help but worth checking (if you have not done so)
<http://assets.nagios.com/downloads/nagiosxi/docs/NRPE-Troubleshooting-and-Common-Solutions.pdf>
and
<http://geekpeek.net/could-not-complete-ssl-handshake/>
— Arun Khan