Problem With SSH Disconnecting

Home » CentOS » Problem With SSH Disconnecting
CentOS 11 Comments

Running CentOS 7 on workstation and having a problem with SSH disconnects. My ssh_config contains:

Host *
TCPKeepAlive yes ServerAliveInterval 30
ServerAliveCountMax 300

and sshd_config on the server contains:

TCPKeepAlive yes ClientAliveInterval 60
ClientAliveCountMax 300

Have I missed any setting needed to prevent these random disconnects? I don’t think there is anything wrong with the network card, the driver, or the cable, since if I am on a VPN connection via another server, the VPN and any SSH connection stay up indefinitely.

Thanks.

11 thoughts on - Problem With SSH Disconnecting

  • There are usually 2 different reasons for this:
    1. The VPN is UDP and times out/drops keeps alives so that they no longer function properly. [The UDP connection will make it look like you have a new SSH connection which of course the system will drop because that would allow for security problems.]

    2. A firewall in the chain of things (system you are on, the system you are going to, or somewhere in between) has session flushing issues. If you have the firewall set up to only accept NEW port 22 connections and then just looks to see if the ESTABLISHED, RELATED tables are accepted elsewhere then if the session somehow ages out or is flushed due to usage, the ssh connection can get dropped.

    The solution to one is to see if a TCP VPN fixes the problem. The second one is to either make the iptables kernel tables larger or to have all port
    22 accepted even if it is not ESTABLISHED.

    These aren’t the only ways the problem you see can occur but they are some of the most common I have run into.

    Stephen J Smoogen.

  • I should also explain that:

    – I am on a workstation (located in the US), ssh-ing into server 1 (located in the US).

    – From server 1 I use scp to transfer large files from server 2 (located in Europe) to server 1 (in the US).

    The above randomly disconnects.

    However, when:

    – I use a VPN connection to server 3 (also located in Europe).

    – From the same workstation as above, do exactly as above, connections are rock-solid.

  • Another poster has provided some possible reasons for the disconnections. Whatever the cause, autossh (from the epel repo) is a good workaround.

  • Not that this happens while I do large scp file transfers that may take more than half an hour, simply restarting an SSH session is not going to help since I will lose the file transfer.

  • I don’t know if this would help but I had a similar issue and it turned out that there was a custom script in /etc/profile.d/ that contain TMOUT 900.

    You can also check in /etc/profile, usually, the security logs has something about the disconnects, of you can use wireshark or a similar tool to capture and analyse the packets.

  • I think that I forgot to mention that the problem is not the common “write failed: broken pipe” but “packet_write_wait: broken pipe”.

    I don’t think the problem is router related since another computer, albeit running CentOS 6, does not have similar issues.

    Does the information above suggest any other reason for my problem?

  • You have two SSH sessions here. AFAICT, it’s the first one
    (workstation->server 1) that randomly disconnects, right?

    It would seem to me that something between your workstation and server
    1 is possibly mishandling TCP options (being it sack, timestamps or both). Is there a router or a firewall in between?

    Try doing a traffic capture between both and see why it hangs. If you’re hesitant to post the binary traffic capture, post the text version of it with anonymized IP addresses. Just please be sure to disable SSH protocol so it would include TCP details on it then.

    Marcelo

  • Let’s reduce the problem to a simpler case: the workstation running CentOS 7 is behind a router, behind the same router is the server running CentOS 6. For various reasons, I access the server via its external IP address, not the local one. Access is via ssh.

    When using straight ssh, the connection times out after random times, always less than 20 minutes or so, though, with the error message “packet_write_wait: broken pipe”, not the common “write failed: broken pipe”.

    However, when I am in a SSH session on a VPN connection (which terminates in an outside server far away from my network), the SSH connection does /not/ time out.

    Further, accessing the same server as first mentioned, in the identical fashion, from another computer running CentOS 6, I never encounter the “packet_write_wait” error and disconnection.

    My conclusion is thus that this is not a router issue but could conceivable be a bug in CentOS 7. I am also ruling out a cable issue or a physical network card issue since the VPN connection is stable.