Connection reset by server after SSH2_MSG_KEXINIT sent

[asterikx]

"OpenSSH for Windows" version
v1.0.0.0-Beta

Server OperatingSystem
Windows 10 Pro

Client OperatingSystem
MacOS High Sierra 10.13.2

What is failing
The connection is reset by the server after SSH2_MSG_KEXINIT sent by the client. I needed RSA support, that's why I switched from the built-in version (which was working) to v1.0.0.0-Beta. I followed the release notes and installation instructions.

Expected output
An established connection/SSH shell.

Actual output
Client-side:

<LOCAL MACHINE>:~ <LOCAL USER>$ ssh <SERVER USER>@<SERVER DOMAIN> -p 40544 -vvv
OpenSSH_7.6p1, LibreSSL 2.6.2
debug1: Reading configuration data /Users/<LOCAL USER>/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to <SERVER DOMAIN> port 40544.
debug1: Connection established.
debug1: identity file /Users/<LOCAL USER>/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<LOCAL USER>/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<LOCAL USER>/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<LOCAL USER>/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<LOCAL USER>/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<LOCAL USER>/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<LOCAL USER>/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<LOCAL USER>/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.6
debug1: match: OpenSSH_7.6 pat OpenSSH* compat 0x04000000
debug3: fd 5 is O_NONBLOCK
debug1: Authenticating to <SERVER DOMAIN>:40544 as '<SERVER USER>'
debug3: put_host_port: [<SERVER DOMAIN>]:40544
debug3: hostkeys_foreach: reading file "/Users/<LOCAL USER>/.ssh/known_hosts"
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
Connection reset by <SERVER PUBLIC IP> port 40544
<LOCAL MACHINE>:~ <LOCAL USER>$ 

Server-side:

C:\Program Files\OpenSSH>sshd.exe -ddd
debug2: load_server_config: filename __PROGRAMDATA__\\ssh/sshd_config
debug2: load_server_config: done config len = 195
debug2: parse_server_config: config __PROGRAMDATA__\\ssh/sshd_config len 195
debug3: __PROGRAMDATA__\\ssh/sshd_config:25 setting LogLevel DEBUG3
debug3: __PROGRAMDATA__\\ssh/sshd_config:39 setting AuthorizedKeysFile .ssh/authorized_keys
debug3: __PROGRAMDATA__\\ssh/sshd_config:53 setting PasswordAuthentication yes
debug3: __PROGRAMDATA__\\ssh/sshd_config:78 setting Subsystem sftp      sftp-server.exe
debug1: sshd version OpenSSH_7.6, LibreSSL 2.5.3
debug1: private host key #0: ssh-rsa SHA256:K7c/7Bdh/fcE/Ldb0I0REXxJ8LWbLfMLDYgBJGboVLE
debug1: private host key #1: ssh-dss SHA256:CSRd/ulPsEqftkWTEXhkzh2mRM/AerSBwrOVHqD+1Zk
debug1: private host key #2: ecdsa-sha2-nistp256 SHA256:SYsWLu8BRugZDIC4YxVY2bPkn4drfd8CS9pNQoX2ebc
debug1: private host key #3: ssh-ed25519 SHA256:VSh1cYltJmCYeS6Xg2i6LFK6PCYzCbvMoJSfeQ0yAHA
debug1: rexec_argv[0]='sshd.exe'
debug1: rexec_argv[1]='-ddd'
debug2: fd 3 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 3 IPV6_V6ONLY
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug2: fd 4 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug3: fd 5 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 8 config len 195
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
Connection from <CLIENT PUBLIC IP> port 49687 on <SERVER LOCAL IP> port 22
debug1: Client protocol version 2.0; client software version OpenSSH_7.6
debug1: match: OpenSSH_7.6 pat OpenSSH* compat 0x04000000
debug1: Local version string SSH-2.0-OpenSSH_7.6
debug2: fd 5 setting O_NONBLOCK
debug3: LsaLogonUser succeeded
debug3: spawning "C:\\Program Files\\OpenSSH\\sshd.exe" "-ddd" "-y"
debug3: send_rexec_state: entering fd = 4 config len 195
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug3: ssh_msg_send: type 0
debug3: ssh_msg_send: type 0
debug3: preauth child monitor started
debug1: monitor_read_log: child log fd closed
debug3: mm_request_receive entering
debug1: do_cleanup

[bagajjal]

Are you running Sshd debug mode as system? If not please see the release notes

[asterikx]

Yes, I did run it as system

C:\Program Files\OpenSSH>whoami
nt authority\system

[manojampalam]

@asterikx
Thanks for reporting this. Apparently, when psexec is used with -i switch, it is causing the stderr redirection issues with sshd that's causing the unprivileged worker to terminate abruptly.

As a workaround, skip -i switch for now (I've updated wiki and trouble shooting pages accordingly)
psexec -s cmd

Please regnerate sshd -d output within the above SYSTEM console.

[asterikx]

Thanks @manojampalam. This indeed gives me more information on what went wrong, but still I can't figure out the causing issue.

Server-side output:

C:\Program Files\OpenSSH>sshd.exe -ddd
debug2: load_server_config: filename __PROGRAMDATA__\\ssh/sshd_config
debug2: load_server_config: done config len = 168
debug2: parse_server_config: config __PROGRAMDATA__\\ssh/sshd_config len 168
debug3: __PROGRAMDATA__\\ssh/sshd_config:25 setting LogLevel DEBUG3
debug3: __PROGRAMDATA__\\ssh/sshd_config:39 setting AuthorizedKeysFile .ssh/authorized_keys
debug3: __PROGRAMDATA__\\ssh/sshd_config:77 setting Subsystem sftp sftp-server.exe
debug1: sshd version OpenSSH_7.6, LibreSSL 2.5.3
debug1: private host key #0: ssh-rsa SHA256:K7c/7Bdh/fcE/Ldb0I0REXxJ8LWbLfMLDYgBJGboVLE
debug1: private host key #1: ssh-dss SHA256:CSRd/ulPsEqftkWTEXhkzh2mRM/AerSBwrOVHqD+1Zk
debug1: private host key #2: ecdsa-sha2-nistp256 SHA256:SYsWLu8BRugZDIC4YxVY2bPkn4drfd8CS9pNQoX2ebc
debug1: private host key #3: ssh-ed25519 SHA256:VSh1cYltJmCYeS6Xg2i6LFK6PCYzCbvMoJSfeQ0yAHA
debug1: rexec_argv[0]='sshd.exe'
debug1: rexec_argv[1]='-ddd'
debug2: fd 3 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 3 IPV6_V6ONLY
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug2: fd 4 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug3: fd 5 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 8 config len 168
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
Connection from <CLIENT PUBLIC IP> port 49833 on <SERVER LOCAL IP> port 22
debug1: Client protocol version 2.0; client software version OpenSSH_7.6
debug1: match: OpenSSH_7.6 pat OpenSSH* compat 0x04000000
debug1: Local version string SSH-2.0-OpenSSH_7.6
debug2: fd 5 setting O_NONBLOCK
unable to generate token for user sshd
unable to generate token on 2nd attempt for user sshd
unable to get security token for user sshd
posix_spawn failed
debug3: send_rexec_state: entering fd = 4 config len 168
debug3: ssh_msg_send: type 0
debug3: write ERROR from cb(2):232, io:000001CDDCA18640
ssh_msg_send: write
send_rexec_state: ssh_msg_send failed
debug1: do_cleanup

[manojampalam]

In this case, sshd (privilege separation account) token couldn't be generated.

I would double check that you are running as SYSTEM.

If yes, please check Troubleshooting wiki and get debug3 traces from sshd service

[asterikx]

I was, checked it by running whoami just beofre `sshd.exe -ddd``

But anyways, I solved my issues, by removing C: \Program Files\OpenSSH and reinstalling. This time I extracted the archive via the Powershell (as admin). (Previously I did it via the GUI and as the currently logged in user. I adjusted the permissions to only include SYSTEM and the Administrators group (FullControl) but maybe they were corrupted nevertheless)

For future reference, here all the commands I run for removing the current installation and reinstalling (assuming the downloaded archive is located at D:\Downlaods):

powershell.exe -ExecutionPolicy Bypass -File 'C:\Program Files\OpenSSH\uninstall-sshd.ps1'
del 'C:\Program Files\OpenSSH\'
Expand-Archive D:\Downloads\OpenSSH-Win64.zip -DestinationPath 'C:\Program Files\OpenSSH'
copy 'C:\Program Files\OpenSSH\OpenSSH-Win64\*' 'C:\Program Files\OpenSSH\'
del 'C:\Program Files\OpenSSH\OpenSSH-Win64\'
powershell.exe -ExecutionPolicy Bypass -File 'C:\Program Files\OpenSSH\install-sshd.ps1'
powershell.exe -ExecutionPolicy Bypass -File 'C:\Program Files\OpenSSH\install-sshd.ps1'
powershell.exe -ExecutionPolicy Bypass -File 'C:\Program Files\OpenSSH\FixHostFilePermissions.ps1'
powershell.exe -ExecutionPolicy Bypass -File 'C:\Program Files\OpenSSH\FixUserFilePermissions.ps1'
Set-Service sshd -StartupType Automatic
Set-Service ssh-agent -StartupType Automatic

After that the permissions on the C: \Program Files\OpenSSH folder were:

icacls 'C:\Program Files\OpenSSH\'
      . NT SERVICE\TrustedInstaller:(I)(F)
        NT SERVICE\TrustedInstaller:(I)(CI)(IO)(F)
        NT AUTHORITY\SYSTEM:(I)(F)
        NT AUTHORITY\SYSTEM:(I)(OI)(CI)(IO)(F)
        BUILTIN\Administrators:(I)(F)
        BUILTIN\Administrators:(I)(OI)(CI)(IO)(F)
        BUILTIN\Users:(I)(RX)
        BUILTIN\Users:(I)(OI)(CI)(IO)(GR,GE)
        CREATOR OWNER:(I)(OI)(CI)(IO)(F)
        APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES:(I)(RX)
        APPLICATION PACKAGE AUTHORITY\ALL APPLICATION PACKAGES:(I)(OI)(CI)(IO)(GR,GE)
        APPLICATION PACKAGE AUTHORITY\ALL RESTRICTED APPLICATION PACKAGES:(I)(RX)
        APPLICATION PACKAGE AUTHORITY\ALL RESTRICTED APPLICATION PACKAGES:(I)(OI)(CI)(IO)(GR,GE)

@manojampalam Maybe update the installation instructions? Previously I removed all permissions except for NT AUTHORITY\SYSTEM and BUILTIN\Administrators which were set to FullControl. I assume it's there where I messed it up.

[ronindesign]

Just wasted 2 hours on this.

[tecfu]

Same problem. Emphasis on this issue in the installation docs recommended.

[manojampalam]

Sorry about that. I've added a reference to this thread in the troubleshooting page. https://github.com/PowerShell/Win32-OpenSSH/wiki/Troubleshooting-Steps

[clintoncampbell]

Having just lost almost 6 hours over two days on this issue (and having overlooked the small note on the troubleshooting page), I'm wondering if there is more to be done than highlight in the Troubleshooting-Steps. Ideally this problem could be detected and fixed automatically via the Install script or FixHostFilePermissions.ps1 script?

Likewise, some indication of the problem in debug-level logging would be helpful. In my case, the DEBUG3 logs showed no indication of anything unusual when sshd was run as a service but connections would immediately reset. Likewise, the problem wouldn't manifest when running sshd directly in debug mode.

The crucial difference? My original script expanded the archive and moved the "OpenSSH-Win64" directory that had been at the root of the archive into "Program Files". This results in entirely different permissions than when I expand the archive into a destination directory in "Program Files" and then move the contents of the "OpenSSH-Win64" subdirectory up to the parent.