Fail to RemoteForward a unix domain socket

[BusyJay]

"OpenSSH for Windows" version

7.7.2.1

Server OperatingSystem

Linux

Client OperatingSystem

Windows 10 Pro

What is failing

I'm trying to forward my local gpg-agent to remote server. Same configuration works fine when both ends are Linux. When trying to forward from Windows to Linux, it doesn't work anymore.
There are three kind of errors.

1. When using configuration RemoteForward /run/user/1000/gnupg/S.gpg-agent ~/AppData/Roaming/gnupg/S.gpg-agent.extra, client reports forward success, but it won't work when actually testing it.

   debug1: remote forward success for: listen /run/user/1000/gnupg/S.gpg-agent:-2, connect ~/AppData/Roaming/gnupg/S.gpg-agent.extra:-2
   debug1: client_input_channel_open: ctype forwarded-streamlocal@openssh.com rchan 3 win 2097152 max 32768
   debug1: client_request_forwarded_streamlocal: /run/user/1000/gnupg/S.gpg-agent
   debug2: fd 8 setting O_NONBLOCK
   debug3: unable to connect to pipe ~/AppData/Roaming/gnupg/S.gpg-agent.extra, error: 3
   debug1: connect_next: host ~/AppData/Roaming/gnupg/S.gpg-agent.extra ([unix]:~/AppData/Roaming/gnupg/S.gpg-agent.extra): No such file or directory
   connect to ~/AppData/Roaming/gnupg/S.gpg-agent.extra port -2 failed: No such file or directory
   debug1: failure forwarded-streamlocal@openssh.com
   

   I suspect it's because CreateFileW can't expand charater ~ to home directory.

2. When configuring using RemoteForward /run/user/1000/gnupg/S.gpg-agent C:/Users/me/AppData/Roaming/gnupg/S.gpg-agent.extra, client reports parse error: Bad forwarding specification.

   I guess configuration parser just can't stand :.

3. When configuring using RemoteForward /run/user/1000/gnupg/S.gpg-agent /Users/me/AppData/Roaming/gnupg/S.gpg-agent.extra, and execute in disk C, client finally build up the channel successfully, but fails when handling connection.

   debug1: remote forward success for: listen /run/user/1000/gnupg/S.gpg-agent:-2, connect /Users/me/AppData/Roaming/gnupg/S.gpg-agent.extra:-2
   debug1: client_input_channel_open: ctype forwarded-streamlocal@openssh.com rchan 3 win 2097152 max 32768
   debug1: client_request_forwarded_streamlocal: /run/user/1000/gnupg/S.gpg-agent
   debug2: fd 8 setting O_NONBLOCK
   debug1: connect_next: host /Users/me/AppData/Roaming/gnupg/S.gpg-agent.extra ([unix]:/Users/me/AppData/Roaming/gnupg/S.gpg-agent.extra) in progress, fd=8
   debug3: fd 8 is O_NONBLOCK
   debug3: fd 8 is O_NONBLOCK
   debug1: channel 1: new [forwarded-streamlocal]
   debug1: confirm forwarded-streamlocal@openssh.com
   debug3: channel 1: waiting for connection
   debug3: w32_getsockopt ERROR: not sock :2
   getsockopt SO_ERROR failed
   debug1: channel 1: connection failed: Not a socket
   connect_to /Users/me/AppData/Roaming/gnupg/S.gpg-agent.extra port -2: failed.
   

   That's because w32_getsockopt rejects work on a unix domain socket, it seems like a bug.

I guess there is no way to get around this now. Any suggestions?

Expected output
gpg-agent is forwarded successfully.

Actual output
It fails.

Editted:
For those who wants to forward gpg agent like me, I wrote a tool that built a bridge between GnuPG and openssh.

[WSLUser]

Well at a first step, try upgrading to 8.1 and attempt to reproduce the issue. Submit updated logs if it still appears.

[BusyJay]

Turns out even OpenSSH for Windows supports UDS, it's still not enough. Because GnuPG on windows does not use UDS at all. The file is a simulated UDS.

For those who wants to forward gpg agent like me, I wrote a tool that built a bridge between GnuPG and openssh. It has been working fine for a month now.

[PeterStaev]

I just got the same problem. I managed to get the path working, but it fails with the same error as OP. This is using v8.1 on Windows 10 Pro (1909). Remote forward is set up like this in the config:

RemoteForward /Users/me/.gnupg/S.gpg-agent C\:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra

debug1: remote forward success for: listen /Users/me/.gnupg/S.gpg-agent:-2, connect C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra:-2
debug1: All remote forwarding requests processed

when trying to run gpg --no-autostart -K

debug3: receive packet: type 90
debug1: client_input_channel_open: ctype forwarded-streamlocal@openssh.com rchan 2 win 2097152 max 32768
debug1: client_request_forwarded_streamlocal: request: /Users/me/.gnupg/S.gpg-agent
debug2: fd 7 setting O_NONBLOCK
debug1: connect_next: host C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra ([unix]:C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra) in progress, fd=7
debug3: fd 7 is O_NONBLOCK
debug3: fd 7 is O_NONBLOCK
debug1: channel 1: new [forwarded-streamlocal]
debug1: confirm forwarded-streamlocal@openssh.com
debug3: channel 1: waiting for connection
debug3: w32_getsockopt ERROR: not sock :2
getsockopt SO_ERROR failed
debug1: channel 1: connection failed: Not a socket
connect_to C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra port -2: failed.
debug3: send packet: type 92
debug2: channel 1: zombie
debug2: channel 1: garbage collecting
debug1: channel 1: free: forwarded-streamlocal, nchannels 2
debug3: channel 1: status: The following connections are open:
  #0 client-session (t4 r1 i0/0 o0/0 e[write]/0 fd 4/5/6 sock -1 cc -1)

gpg: can't connect to the agent: End of file
gpg: no gpg-agent running in this session
debug3: receive packet: type 90
debug1: client_input_channel_open: ctype forwarded-streamlocal@openssh.com rchan 2 win 2097152 max 32768
debug1: client_request_forwarded_streamlocal: request: /Users/me/.gnupg/S.gpg-agent
debug2: fd 7 setting O_NONBLOCK
debug1: connect_next: host C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra ([unix]:C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra) in progress, fd=7
debug3: fd 7 is O_NONBLOCK
debug3: fd 7 is O_NONBLOCK
debug1: channel 1: new [forwarded-streamlocal]
debug1: confirm forwarded-streamlocal@openssh.com
debug3: channel 1: waiting for connection
debug3: w32_getsockopt ERROR: not sock :2
getsockopt SO_ERROR failed
debug1: channel 1: connection failed: Not a socket
connect_to C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra port -2: failed.
debug3: send packet: type 92
debug2: channel 1: zombie
debug2: channel 1: garbage collecting
debug1: channel 1: free: forwarded-streamlocal, nchannels 2
debug3: channel 1: status: The following connections are open:
  #0 client-session (t4 r1 i0/0 o0/0 e[write]/0 fd 4/5/6 sock -1 cc -1)

gpg: can't connect to the agent: End of file
debug3: receive packet: type 90
debug1: client_input_channel_open: ctype forwarded-streamlocal@openssh.com rchan 2 win 2097152 max 32768
debug1: client_request_forwarded_streamlocal: request: /Users/me/.gnupg/S.gpg-agent
debug2: fd 7 setting O_NONBLOCK
debug1: connect_next: host C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra ([unix]:C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra) in progress, fd=7
debug3: fd 7 is O_NONBLOCK
debug3: fd 7 is O_NONBLOCK
debug1: channel 1: new [forwarded-streamlocal]
debug1: confirm forwarded-streamlocal@openssh.com
debug3: channel 1: waiting for connection
debug3: w32_getsockopt ERROR: not sock :2
getsockopt SO_ERROR failed
debug1: channel 1: connection failed: Not a socket
connect_to C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra port -2: failed.
debug3: send packet: type 92
debug2: channel 1: zombie
debug2: channel 1: garbage collecting
debug1: channel 1: free: forwarded-streamlocal, nchannels 2
debug3: channel 1: status: The following connections are open:
  #0 client-session (t4 r1 i0/0 o0/0 e[write]/0 fd 4/5/6 sock -1 cc -1)

debug3: receive packet: type 90
debug1: client_input_channel_open: ctype forwarded-streamlocal@openssh.com rchan 2 win 2097152 max 32768
debug1: client_request_forwarded_streamlocal: request: /Users/me/.gnupg/S.gpg-agent
debug2: fd 7 setting O_NONBLOCK
debug1: connect_next: host C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra ([unix]:C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra) in progress, fd=7
debug3: fd 7 is O_NONBLOCK
debug3: fd 7 is O_NONBLOCK
debug1: channel 1: new [forwarded-streamlocal]
debug1: confirm forwarded-streamlocal@openssh.com
debug3: channel 1: waiting for connection
gpg: can't connect to the agent: End of file
debug3: w32_getsockopt ERROR: not sock :2
getsockopt SO_ERROR failed
debug1: channel 1: connection failed: Not a socket
connect_to C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra port -2: failed.
debug3: send packet: type 92
debug2: channel 1: zombie
debug2: channel 1: garbage collecting
debug1: channel 1: free: forwarded-streamlocal, nchannels 2
debug3: channel 1: status: The following connections are open:
  #0 client-session (t4 r1 i0/0 o0/0 e[write]/0 fd 4/5/6 sock -1 cc -1)

gpg: can't connect to the agent: End of file
debug3: receive packet: type 90
debug1: client_input_channel_open: ctype forwarded-streamlocal@openssh.com rchan 2 win 2097152 max 32768
debug1: client_request_forwarded_streamlocal: request: /Users/me/.gnupg/S.gpg-agent
debug2: fd 7 setting O_NONBLOCK
debug1: connect_next: host C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra ([unix]:C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra) in progress, fd=7
debug3: fd 7 is O_NONBLOCK
debug3: fd 7 is O_NONBLOCK
debug1: channel 1: new [forwarded-streamlocal]
debug1: confirm forwarded-streamlocal@openssh.com
debug3: channel 1: waiting for connection
debug3: w32_getsockopt ERROR: not sock :2
getsockopt SO_ERROR failed
debug1: channel 1: connection failed: Not a socket
connect_to C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra port -2: failed.
debug3: send packet: type 92
debug2: channel 1: zombie
debug2: channel 1: garbage collecting
debug1: channel 1: free: forwarded-streamlocal, nchannels 2
debug3: channel 1: status: The following connections are open:
  #0 client-session (t4 r1 i0/0 o0/0 e[write]/0 fd 4/5/6 sock -1 cc -1)

gpg: can't connect to the agent: End of file
debug3: receive packet: type 90
debug1: client_input_channel_open: ctype forwarded-streamlocal@openssh.com rchan 2 win 2097152 max 32768
debug1: client_request_forwarded_streamlocal: request: /Users/me/.gnupg/S.gpg-agent
debug2: fd 7 setting O_NONBLOCK
debug1: connect_next: host C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra ([unix]:C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra) in progress, fd=7
debug3: fd 7 is O_NONBLOCK
debug3: fd 7 is O_NONBLOCK
debug1: channel 1: new [forwarded-streamlocal]
debug1: confirm forwarded-streamlocal@openssh.com
debug3: channel 1: waiting for connection
debug3: w32_getsockopt ERROR: not sock :2
getsockopt SO_ERROR failed
debug1: channel 1: connection failed: Not a socket
connect_to C:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra port -2: failed.
debug3: send packet: type 92
debug2: channel 1: zombie
debug2: channel 1: garbage collecting
debug1: channel 1: free: forwarded-streamlocal, nchannels 2
debug3: channel 1: status: The following connections are open:
  #0 client-session (t4 r1 i0/0 o0/0 e[write]/0 fd 4/5/6 sock -1 cc -1)

gpg: can't connect to the agent: End of file

[maertendMSFT]

@BusyJay , can you share your steps for this solution

[BusyJay]

You mean how does the tool work around the problem? The simulated socket used by gnupg stores the port of its http service. So the tool just read the port and build a local http redirect service bridging between gnupg and ssh.

The steps about how to use the tool is written in README: https://github.com/BusyJay/gpg-bridge.

[idarlund]

I'm so close in using Yubikey+gpg to admin linux servers over windows' built in openssh client! This agent forwarding is the only missing bit so that I can forward my gpg agent (and yubikey) to jump hosts to be able to manage other linux hosts.

Microsoft: Please fix this issue ❤️

[c3c]

Running into the same issue. +1

[nezorflame]

Few months later and I too got the same issue here, with the same conditions and the same setup.

OpenSSH version: OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2

Regarding this:

I just got the same problem. I managed to get the path working, but it fails with the same error as OP. This is using v8.1 on Windows 10 Pro (1909). Remote forward is set up like this in the config:

RemoteForward /Users/me/.gnupg/S.gpg-agent C\:\\Users\\me\\AppData\\Roaming\\gnupg/S.gpg-agent.extra

you could just change the : to \: and leave the other slashes as is like this:

RemoteForward /run/user/1000/gnupg/S.gpg-agent C\:/Users/me/AppData/Roaming/gnupg/S.gpg-agent.extra

[MathiasMagnus]

I did the same hack to escape @nezorflame suggested and it seems that the agent is connected.

ssh -vvv my_remote_end
...
debug1: Remote connections from /run/user/1001/gnupg/S.gpg-agent:-2 forwarded to local address C:/Users/mate/AppData/Roaming/gnupg/S.gpg-agent.extra:-2
...
debug1: remote forward success for: listen /run/user/1001/gnupg/S.gpg-agent:-2, connect C:/Users/mate/AppData/Roaming/gnupg/S.gpg-agent.extra:-2
debug1: All remote forwarding requests processed
...

Also checking if the agent could be reconnected after SSH connection has been established is successful (as suggested here):

gpg-connect-agent /bye
echo $?
0

Also what @PeterStaev posted as being a problem also seems successful (at least to me):

gpg --no-autostart -K
/home/mate/.gnupg/pubring.kbx
-----------------------------
sec   rsa4096 2019-09-10 [SC]
      ...
uid           [ unknown] My Name <my@email.com>
ssb   rsa4096 2019-09-10 [E]

However, when I try to sign something, pinentry-qt doesn't pop up on my local box, but bails out with missing ioctl device.

echo "" | gpg --clearsign
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512


gpg: signing failed: Inappropriate ioctl for device
gpg: [stdin]: clear-sign failed: Inappropriate ioctl for device

I know that one solution is to att export GPG_TTY=$(tty) however that defeats the entire purpose of agent forwarding. I don't want to deploy my private keys on the remote end, neither do I want to be prompted on the command-line for passwords, because I want to use the Git GUI of VS Code from a remote session and I want the password prompt to pop up on my local machine running Windows. gpg.exe locally pops open pinentry-qt as expected when prompting for a password. How can I tell the remote gpg program to tunnel through the agent to hit my local pinentry program?

[m1cr0man]

Hey, would just like to bump this as Google is leading me here in my searches.

I have git-bash installed too (Not in WSL, just on Windows itself) which ships with OpenSSH 8.4. Using either a C\:/Users/... or /c/Users/... style path in my ssh config, I am able to get a 100% working agent forward set up to a remote Linux host with no real hassle. Using that binary is working in all the local shells I've tried too, those being Powershell, cmd.exe, and Git Bash.

For my use case, all I really wanted was working forwarding within VS Code Remote, so I changed remote.SSH.path to C:\Program Files\Git\usr\bin\ssh.exe and now I can sign things :) The down side is I must give up using the OpenSSH Auth Agent service which remembers my key passphrase, but that's a sacrifice I'm willing to make.

I'm still confused as to why it would work on the the basic OpenSSH implementation and not OpenSSH For Windows? Are they not the same, at least in regards to how forwarding is handled?

My versions:

PS C:\> ssh -V
OpenSSH_for_Windows_8.1p1, LibreSSL 3.0.2
PS C:\Program Files\Git\usr\bin> .\ssh -V
OpenSSH_8.4p1, OpenSSL 1.1.1h  22 Sep 2020

[hoang-himself]

Is there any update on this?

I did some mix and match with Git for Windows's SSH and GPG¹, with Win32-OpenSSH and Gpg4win¹. Only Git for Windows bundle works, any other combination fails.

In the process I also noticed that StreamLocalBindUnlink does not work if socket forwarding fails for some reason.

Footnotes

1. I mean GPG extra socket ↩ ↩²

[giggio]

I got this to work in WSL2, and I believe I have a clue of what is happening and why this does not work on Windows.
The $env:LOCALAPPDATA/gnupg/S.gpg-agent and $env:LOCALAPPDATA/gnupg/S.gpg-agent.extra files on Windows are not really Unix sockets. And ssh is able to forward only Unix sockets, AFAIK. So this configuration will not work with Windows SSH until Windows OpenSSH implementation supports Unix Sockets. Windows already supports them.

This might explain why some people have been able to make it work with git ssh, it is probably emulating Unix sockets somehow, or even using the new and now available AF Unix Socket support.

So, how do you get it to work with WSL2? I'm using wsl-relay: https://github.com/Lexicality/wsl-relay

All I need to do is (in WSL):

socat UNIX-LISTEN:$HOME/.gnupg/S.gpg-agent.extra,fork, EXEC:'wsl-relay.exe --input-closes --pipe-closes --gpg=S.gpg-agent.extra',nofork

And in the ssh config file (in WSL):

RemoteForward /run/user/1000/gnupg/S.gpg-agent /home/<user>/.gnupg/S.gpg-agent.extra  

And it works.
See how I wired it automatically so I don't have to run socat manually here:
https://github.com/giggio/bashscripts/blob/8ba5ac02849904e787cb51b4c607e9858c12baf7/bashrc-wsl.bash#L109

[idarlund]

WSL/WSL2 is not the issue. We want it to be supported in the native Windows OpenSSH client 🙂
Microsoft: Could you please prioritize this issue? ❤️