SSH login hang when working from home

I have had a really annoying issue with ssh sessions when working from home on my ubuntu system. I narrowed it down to my BT HomeHub 5 router but didn’t look into the ‘exact’ root cause, my bad! Everything worked perfectly over my 4G phone connection 🙂 I just needed the slightest excuse to justify buying a new router and moving away from the restrictive HomeHub.
I settled upon the TP-Link Archer VR2600 – great reviews at the time however……I had the exact same issue.

So if anyone else is have issues logging in from home (or a NAT’d location) using OpenSSH and you get the following error ‘hang’ when logging in –

grazzer@grazzer-dev:~$ ssh -vvv -i Dev/DevOpsPipeline_SSH_Keypair.pem ubuntu@62.60.47.230
OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /home/grazzer/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "62.60.47.230" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 62.60.47.230 [62.60.47.230] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file Dev/DevOpsPipeline_SSH_Keypair.pem type -1
debug1: key_load_public: No such file or directory
debug1: identity file Dev/DevOpsPipeline_SSH_Keypair.pem-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 62.60.47.230:22 as 'ubuntu'
debug3: hostkeys_foreach: reading file "/home/grazzer/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/grazzer/.ssh/known_hosts:9
debug3: load_hostkeys: loaded 1 keys from 62.60.47.230
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:kNUzPcnWcJRBu/2WeSt7L/CVg2aQE3AXLnnF7jSJL88
debug3: hostkeys_foreach: reading file "/home/grazzer/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/grazzer/.ssh/known_hosts:9
debug3: load_hostkeys: loaded 1 keys from 62.60.47.230
debug1: Host '62.60.47.230' is known and matches the ECDSA host key.
debug1: Found key in /home/grazzer/.ssh/known_hosts:9
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug2: key: allthingsclowd@users.noreply.github.com (0xd578e6eaa0), agent
debug2: key: grazzer@grazzer-dev (0xd578e6f4a0), agent
debug2: key: Dev/DevOpsPipeline_SSH_Keypair.pem ((nil)), explicit
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: allthingsclowd@users.noreply.github.com
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Offering RSA public key: grazzer@grazzer-dev
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Trying private key: Dev/DevOpsPipeline_SSH_Keypair.pem
debug3: sign_and_send_pubkey: RSA SHA256:qgidm4M5bU16GRqFo03OEyWSaAo7FVmoeon/XypwC3A
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to 62.60.47.230 ([62.60.47.230]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 91
debug2: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x10
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug3: send packet: type 98
debug1: Sending environment.
debug3: Ignored env XDG_VTNR
debug3: Ignored env XDG_SESSION_ID
debug3: Ignored env XDG_GREETER_DATA_DIR
debug3: Ignored env CLUTTER_IM_MODULE
debug3: Ignored env SESSION
debug3: Ignored env GPG_AGENT_INFO
debug3: Ignored env TERM
debug3: Ignored env VTE_VERSION
debug3: Ignored env SHELL
debug3: Ignored env QT_LINUX_ACCESSIBILITY_ALWAYS_ON
debug3: Ignored env BYOBU_CONFIG_DIR
debug3: Ignored env WINDOWID
debug3: Ignored env UPSTART_SESSION
debug3: Ignored env GNOME_KEYRING_CONTROL
debug3: Ignored env GTK_MODULES
debug3: Ignored env BYOBU_READLINK
debug3: Ignored env USER
debug3: Ignored env LS_COLORS
debug3: Ignored env QT_ACCESSIBILITY
debug3: Ignored env BYOBU_RUN_DIR
debug3: Ignored env XDG_SESSION_PATH
debug3: Ignored env XDG_SEAT_PATH
debug3: Ignored env BYOBU_DISTRO
debug3: Ignored env DESKTOP_MODE
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env BYOBU_DATE
debug3: Ignored env DEFAULTS_PATH
debug3: Ignored env XDG_CONFIG_DIRS
debug3: Ignored env BYOBU_SED
debug3: Ignored env BYOBU_BACKEND
debug3: Ignored env DESKTOP_SESSION
debug3: Ignored env PATH
debug3: Ignored env QT_IM_MODULE
debug3: Ignored env QT_QPA_PLATFORMTHEME
debug3: Ignored env BYOBU_DARK
debug3: Ignored env XDG_SESSION_TYPE
debug3: Ignored env PWD
debug3: Ignored env JOB
debug3: Ignored env XMODIFIERS
debug3: Ignored env BYOBU_ULIMIT
debug3: Ignored env GNOME_KEYRING_PID
debug1: Sending env LANG = en_GB.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env GDM_LANG
debug3: Ignored env MANDATORY_PATH
debug3: Ignored env NODE_PATH
debug3: Ignored env COMPIZ_CONFIG_PROFILE
debug3: Ignored env IM_CONFIG_PHASE
debug3: Ignored env BYOBU_WINDOW_NAME
debug3: Ignored env BYOBU_PYTHON
debug3: Ignored env GDMSESSION
debug3: Ignored env SESSIONTYPE
debug3: Ignored env GTK2_MODULES
debug3: Ignored env SHLVL
debug3: Ignored env HOME
debug3: Ignored env XDG_SEAT
debug3: Ignored env LANGUAGE
debug3: Ignored env BYOBU_LIGHT
debug3: Ignored env GNOME_DESKTOP_SESSION_ID
debug3: Ignored env UPSTART_INSTANCE
debug3: Ignored env UPSTART_EVENTS
debug3: Ignored env XDG_SESSION_DESKTOP
debug3: Ignored env LOGNAME
debug3: Ignored env COMPIZ_BIN_PATH
debug3: Ignored env DBUS_SESSION_BUS_ADDRESS
debug3: Ignored env BYOBU_PAGER
debug3: Ignored env XDG_DATA_DIRS
debug3: Ignored env QT4_IM_MODULE
debug3: Ignored env BYOBU_ACCENT
debug3: Ignored env LESSOPEN
debug3: Ignored env BYOBU_PREFIX
debug3: Ignored env INSTANCE
debug3: Ignored env UPSTART_JOB
debug3: Ignored env XDG_RUNTIME_DIR
debug3: Ignored env DISPLAY
debug3: Ignored env XDG_CURRENT_DESKTOP
debug3: Ignored env BYOBU_HIGHLIGHT
debug3: Ignored env GTK_IM_MODULE
debug3: Ignored env LESSCLOSE
debug3: Ignored env BYOBU_TIME
debug3: Ignored env XAUTHORITY
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768

This is caused by OpenSSH setting the TOS (type of service) packet header once authentication is successful and MANY home routers ‘choke’ when they receive this packet. Very disappointed that the VR2600 can’t manage this out-of-the-box.

Workaround 1 (general): Proxy all ssh traffic through netcat (nc) which doesn’t set the TOS datagram.

ssh -o "ProxyCommand nc %h %p" -i Dev/DevOpsPipeline_SSH_Keypair.pem ubuntu@62.60.47.230

You could also add this to your ssh config file (~/.ssh/config) to save repeatedly typing it in. Add a section with the following –

# Directive to overcome TOS issue with our NAT router. During session setup,
# OpenSSH sets the TOS (type of service) field after the user has submitted
# the password. Some routers are known to choke on this, with the result
# that the session hangs during buildup. As workaround we send our traffic
# via netcat which doesn't set the TOS field.
ProxyCommand nc %h %p

Workaround 2 (TP-Link VR2600 specific):
Enable bandwidth control and this seems to activate QoS which in turn ensures that the router now respects the TOS datagram or is it called a packet header?

Hope this helps others and saves some debug time.

TTFN,
Graham

2 thoughts on “SSH login hang when working from home”

  1. Thank you so much for posting this solution. I had the same issue when trying to ssh from a Raspberry Pi 3 to a remote server. The first solution worked for me.

    Strangely I didn’t have the problem when ssh’ing to the same remote server from my laptop (Macbook) (the one from which I was ssh’ing into the RPi). In my case it does not seem to be a problem with the router. The same “debug2: channel 0: open confirm rwindow 0 rmax 32768” line appears when ssh’ing from my macbook but there the ssh does not hang on that line.

    I think the same issue was also interfering with push git commits to Bitbucket from a Raspberry Pi.

    Thanks!

    Like

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

Gravatar
WordPress.com Logo

You are commenting using your WordPress.com account. ( Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. ( Log Out /  Change )

Cancel

Connecting to %s