Debug stepping via ssh takes >3sec per step

[Meai]

Same problem in both vscode and visual studio.

Setup:
localhost ssh into an ubuntu VM in vmplayer.
dotnet --version 9.0.100 on host
dotnet --version 9.0.200 on linux VM

Problem:
Stepping takes >3sec.
Stepping over a simple Console.WriteLine() or in general takes over 3sec. The following text file is the log output of a single such step, as you can see the tickcount at the start and the tickcount at the end show that it's over 3 seconds. Just My Code is enabled.

debuglog.txt

Expected:
It should be nearly as fast as regular debugging (<0.5sec) on localhost ssh.