vz: add SSH over AF_VSOCK

[norio-nomura]

Description:

Since systemd v256 (Ubuntu 24.10), SSH is bound to AF_VSOCK port 22.

https://github.com/systemd/systemd/releases/tag/v256

• If the system is run in a VM providing AF_VSOCK support, it automatically
  binds sshd to AF_VSOCK port 22.

https://discourse.ubuntu.com/t/oracular-oriole-release-notes/44878

• When sshd is installed on a system, a new systemd generator, systemd-ssh-generator
  binds a socket-activated SSH server to local AF_VSOCK and AF_UNIX sockets under certain conditions.

This PR changes to delay starting SSH port forwarding until the SSH server on the VM becomes ready. If AF_VSOCK port 22 can be connected, start a local SSH port as a proxy for AF_VSOCK port 22, instead of starting gvisor's port forwarder.

SSH over VSOCK is faster than SSH over gvisor's port forwarder.

This change is opt-out because it requires VZ and VM with systemd v256+,
setting LIMA_SSH_OVER_VSOCK=true does not mean it works.
To disable, set LIMA_SSH_OVER_VSOCK=false.

Benchmark logs:

On MacBook Pro 14 inch, 2023 with Apple M2 Pro

SETUP:

$ sw_vers
ProductName:		macOS
ProductVersion:		15.7
BuildVersion:		24G219
$ limactl start template://ubuntu-24.10 --rosetta --containerd=none --log-level error
<snip>
$ limactl shell ubuntu-24.10 sudo apt-get -U install -y iperf3 -qqq
<snip>

GRPC Port Forwarder (Current):

$ limactl shell ubuntu-24.10 iperf3 -s
WARN[0000] Both top-level 'rosetta' and 'vmOpts.vz.rosetta' are configured. Using vmOpts.vz.rosetta. Top-level 'rosetta' is deprecated. 
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from ::1, port 53668
[  5] local ::1 port 5201 connected to ::1 port 53672
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   354 MBytes  2.97 Gbits/sec                  
[  5]   1.00-2.00   sec   359 MBytes  3.01 Gbits/sec                  
[  5]   2.00-3.00   sec   354 MBytes  2.97 Gbits/sec                  
[  5]   3.00-4.00   sec   354 MBytes  2.97 Gbits/sec                  
[  5]   4.00-5.00   sec   357 MBytes  3.00 Gbits/sec                  
[  5]   5.00-6.00   sec   354 MBytes  2.97 Gbits/sec                  
[  5]   6.00-7.00   sec   355 MBytes  2.98 Gbits/sec                  
[  5]   7.00-8.00   sec   355 MBytes  2.98 Gbits/sec                  
[  5]   8.00-9.00   sec   355 MBytes  2.98 Gbits/sec                  
[  5]   9.00-9.98   sec   352 MBytes  3.01 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-9.98   sec  3.47 GBytes  2.98 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated

SSH Port Forwarder on gvisor's virtual network (Old):

$ LIMA_SSH_PORT_FORWARDER=true LIMA_SSH_OVER_VSOCK=false limactl restart ubuntu-24.10 --log-level error
$ limactl shell ubuntu-24.10 iperf3 -s
WARN[0000] Both top-level 'rosetta' and 'vmOpts.vz.rosetta' are configured. Using vmOpts.vz.rosetta. Top-level 'rosetta' is deprecated. 
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from ::1, port 43814
[  5] local ::1 port 5201 connected to ::1 port 43824
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   277 MBytes  2.32 Gbits/sec                  
[  5]   1.00-2.00   sec   280 MBytes  2.35 Gbits/sec                  
[  5]   2.00-3.00   sec   278 MBytes  2.34 Gbits/sec                  
[  5]   3.00-4.00   sec   279 MBytes  2.34 Gbits/sec                  
[  5]   4.00-5.00   sec   279 MBytes  2.34 Gbits/sec                  
[  5]   5.00-6.00   sec   279 MBytes  2.34 Gbits/sec                  
[  5]   6.00-7.00   sec   281 MBytes  2.36 Gbits/sec                  
[  5]   7.00-8.00   sec   280 MBytes  2.35 Gbits/sec                  
[  5]   8.00-9.00   sec   281 MBytes  2.36 Gbits/sec                  
[  5]   9.00-10.00  sec   279 MBytes  2.34 Gbits/sec                  
[  5]  10.00-10.03  sec  9.00 MBytes  2.34 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.03  sec  2.74 GBytes  2.34 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated

SSH Port Forwarder over AF_VSOCK (New):

$ LIMA_SSH_PORT_FORWARDER=true limactl restart ubuntu-24.10 --log-level error
$ limactl shell ubuntu-24.10 iperf3 -s
WARN[0000] Both top-level 'rosetta' and 'vmOpts.vz.rosetta' are configured. Using vmOpts.vz.rosetta. Top-level 'rosetta' is deprecated. 
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from ::1, port 57942
[  5] local ::1 port 5201 connected to ::1 port 57948
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   422 MBytes  3.54 Gbits/sec                  
[  5]   1.00-2.00   sec   425 MBytes  3.57 Gbits/sec                  
[  5]   2.00-3.00   sec   427 MBytes  3.58 Gbits/sec                  
[  5]   3.00-4.00   sec   426 MBytes  3.58 Gbits/sec                  
[  5]   4.00-5.00   sec   426 MBytes  3.58 Gbits/sec                  
[  5]   5.00-6.00   sec   428 MBytes  3.59 Gbits/sec                  
[  5]   6.00-7.00   sec   426 MBytes  3.58 Gbits/sec                  
[  5]   7.00-8.00   sec   428 MBytes  3.59 Gbits/sec                  
[  5]   8.00-9.00   sec   425 MBytes  3.57 Gbits/sec                  
[  5]   9.00-10.00  sec   425 MBytes  3.57 Gbits/sec                  
[  5]  10.00-10.00  sec  1.62 MBytes  3.83 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  4.16 GBytes  3.57 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated

[AkihiroSuda]

Can be another PR, but wondering if we can further optimize the performance with (an equivalent of) systemd-ssh-proxy — SSH client plugin for connecting to AF_VSOCK and AF_UNIX sockets
https://www.freedesktop.org/software/systemd/man/256/systemd-ssh-proxy.html

[norio-nomura]

I don't understand the meaning of the comment.

[AkihiroSuda]

I thought it might be possible to eliminate the TCP->vsock forwarder and let ssh directly connect to the vsock.

[norio-nomura]

VM's VSOCK should not be open to the host unless the process that is starting the VM is mediated. As this PR is doing.

[norio-nomura]

Ubuntu 24.04 and earlier, SSH over AF_VSOCK can be enabled with:

provision:
- mode: yq
  path: /etc/systemd/system/ssh.socket.d/vsock.conf
  format: ini
  expression: |
    .Socket.ListenStream="vsock::22"

and upgraded SSH tracked at sshd socket activation does not support AF_VSOCK

[norio-nomura]

How to enable SSH over AF_VSOCK on Ubuntu 20.04, 22.04, and 24.04

Ubuntu 20.04, 22.04:

Since the patched version of SSH has already been released, it requires:

• Add ListenStream="vsock::22" to ssh.socket configuration
• Change ssh.service to use socket-based activation

$ limactl start template://ubuntu-20.04 --rosetta --containerd=none --set '.provision|=.//empty + [{
  "mode": "yq",
  "format": "ini",
  "path": "/etc/systemd/system/ssh.socket.d/vsock.conf",
  "expression": ".Socket.ListenStream=\"vsock::22\""
}, {
  "mode": "system",
  "script": ("#!/bin/bash
set -eux -o pipefail
systemctl is-enabled ssh.service || exit 0
# use socket based activation
systemctl disable --now ssh.service
systemctl enable --now ssh.socket
"|. style="literal")
}]'
$ limactl restart ubuntu-20.04 2>&1 |grep -i vsock
time="2025-09-07T17:01:40+09:00" level=info msg="[hostagent] started vsock forwarder: localhost:59291 -> vsock:22 on VM"
time="2025-09-07T17:01:40+09:00" level=info msg="[hostagent] Detected SSH server is listening on the vsock port; changed localhost:59291 to proxy for the vsock port"

Ubuntu 24.04:

Since the patched version of SSH is not yet released (2025/09/07), it requires installing the proposed version.
The patched version of SSH has been released (2025/09/09), update openssh-server to latest release.

• Add ListenStream="vsock::22" to the ssh.socket configuration.
• Install the proposed version of SSH. Update openssh-server to patched version (1:9.6p1-3ubuntu13.14)
• Already, SSH has been changed to default socket-based activation.

$ limactl start template://ubuntu-24.04 --rosetta --containerd=none --set '.provision|=.//empty + [{
  "mode": "yq",
  "format": "ini",
  "path": "/etc/systemd/system/ssh.socket.d/vsock.conf",
  "expression": ".Socket.ListenStream=\"vsock::22\""
}, {
  "mode": "system",
  "script": ("#!/bin/bash
ss -l --vsock|grep \*:22 -q && exit 0
apt-get --update install --assume-yes openssh-server
"|. style="literal")
}]'
$ limactl restart ubuntu-24.04 2>&1 |grep -i vsock
time="2025-09-07T17:01:56+09:00" level=info msg="[hostagent] started vsock forwarder: localhost:59299 -> vsock:22 on VM"
time="2025-09-07T17:01:56+09:00" level=info msg="[hostagent] Detected SSH server is listening on the vsock port; changed localhost:59299 to proxy for the vsock port"

Edit: The patched version of openssh-server has been released on Ubuntu 24.04

[norio-nomura]

I'm considering another PR for the port forwarder implementation to VSOCK, which can be defined in lima.yaml, but I'm not sure if it's better to mix the rules with .portForwards[]. 🤔

[norio-nomura]

Ubuntu 24.04:

Since the patched version of SSH is not yet released (2025/09/07), it requires installing the proposed version.

The patched version of SSH has been released (2025/09/09), update openssh-server to latest release.

[norio-nomura]

To support custom usernet, I will move WaitOpeningSSHPort logic to the server side.

[norio-nomura]

To support custom usernet, I will move WaitOpeningSSHPort logic to the server side.

done.

[norio-nomura]

To support custom usernet, I will move WaitOpeningSSHPort logic to the server side.

This added an entry point /extension/wait_port?ip=<ip address>&port=<port> to endpoint sock of usernet.

[norio-nomura]

Updated to use SSHAddress as the host address.

[norio-nomura]

This line was not effective until this PR, because above defer:

	filesToRemove := make(map[string]struct{})
	defer func() {
		for f := range filesToRemove {
			_ = os.RemoveAll(f)
		}
	}()

has passed before filesToRemove[pidFile] = struct{}{} has effect.
Because I changed to wait waitSSHLocalPortAccessible, above defer changed to remove pidFile.
That’s why test fails with:
https://github.com/lima-vm/lima/actions/runs/17849303510/job/50754678943?pr=3979#step:9:165

time="2025-09-19T05:30:22Z" level=fatal msg="expected status "Running", got "Broken" (maybe use limactl stop -f?)"

I think filesToRemove is not required.
I'll remove them.

[AkihiroSuda]

Thanks

[norio-nomura]

Thanks! 🙏🏻