Cluster workflow feature to allow shell commands or script to run before remote server setup (e.g. slurm) (wrap install script)

I want to be able to connect to our institution's cluster using VS Code Remote SSH without the server running on a compute node instead of the login node. The preferred workflow is to SSH into the login node and then use a command to allocate a job and spin up an interactive shell on a compute node, and then run any further tasks from there. VS Code Remote SSH doesn't appear to have a feature that facilitates this workflow. I want to be able to inject the spin-up command immediately after SSH'ing into the cluster, but before the VS Code server is set up on the cluster, and before any other tasks are run.

I managed to modify the extension.js file in the following way:

CTRL+F -> "bash
Change the string literal "bash" to "bash -c \"MY_COMMAND bash\""

I've confirmed that this correctly starts the VS Code Remote SSH server on a compute node. Now I am running into a port-forwarding issue, possibly related to issue #92. Our compute nodes have the ports used by VS Code Remote SSH disabled, so there isn't an easy way around this issue.

Thanks for the hard work on this so far! This extension has extraordinary potential. Being able to run and modify a Jupyter notebook remotely on our cluster, while using intellisense and gitlens, AND conda environment detection and dynamic swapping, all in a single application for FREE is incredible.

Our compute nodes have the ports used by VS Code Remote SSH disabled, so there isn't an easy way around this issue.

Do you mean that port forwarding for ssh is disabled on that server? Or are you able to forward some other port over an ssh connection to that server?

Port forwarding for SSH is not disabled on any part of our cluster. I am not intentionally attempting to forward any other ports to the server. I was using remote.SSH.enableDynamicForwarding and remote.SSH.useLocalServer. Your questions gave me the idea to disable those options. I can't determine if that has helped because my earlier assertion was incorrect. I can't get the server to run on a compute node.

To address that issue, and to clarify our workflow some, we are using Slurm. It is highly preferred to have tasks running within a job context so that login node resources aren't being consumed. To do that, we create a job using srun (or one of its siblings) with appropriate resource request parameters. Any commands we want to run are provided as the final argument to srun. All calls to srun must have a command, because it uses execve() to invoke the commands apparently. If no command is passed, srun fails with an error message. With that in mind, setting up the VS Code server on the remote would have to be funneled through a call to srun. Any other method of invocation (such as bash -c) will result in commands being run out of the job context, and thus on the login node. Naively modifying the bash invocation does not work, apparently because srun never receives any arguments. It isn't clear to me how the server installer gets invoked and set up, so I can't offer any suggestions.

As a side note, it is also possible to provide the argument --pty bash to srun to get a terminal within the job context on a node allocated for that job. Looking at #1671, specifically here. It seems like it should be possible to adjust the invocation of bash -ilc to do additional things (found by ctrl+f). I've tried testing this but it doesn't look like that code is called at any point that I can tell, using echo for debugging.

What code do you mean by "that code"? I don't think the issue you point to is related.

We run the installer script essentially like echo <installer script here> | ssh hostname bash. There is an old feature request to be able to run a custom script before running the installer. I am not sure whether that would help you here, is there a way with Slurm to run a command, then have the rest of the same script run in a job context?

It sounds more like you need a way to wrap the full installer script in a custom command, like srun "<installer script here>" is that right?

Yes to your last question, ideally with the ability to customize the wrapping command.

This would be a important feature for vscode-remote. I am currently trying to use vscode to run some interactive python code in a shared cluster and the only way of doing it is by using the srun command of slurm. I'll try to find a workaround, but I think there really is a user case for this feature request.

roblourens

I've got the same issue, but with using LSF instead of SLURM.
As roblourens points out here: #1829 (comment)
just running the install script and starting the server only solves half the problem. Once the server is started, I surmise that VS code will still try sshing directly into the desired (login-restricted) machine to discover what port the VS-remote server picked, as well as starting new terminals that show up in the GUI.

Basically, the only way this can work is if all subprocesses for servers and user terminals are strictly forked children from the original seed shell acquired from LSF/SLURM/whatever job manager you are using. A hacky workaround may be to use something like Paramiko to start a mini-SSH server from the seed shell and then login to this mini server directly from VS Code (assuming there isn't a firewall blocking you, but obviously reverse SSH tunnels can be used to get around that).

Hi all, feel free to ask me to be quiet about this in this thread, but I've managed to get my vscode slurm ssh wrapper script working really well now!

It is a wrapper around ssh that you point vscode to instead of your ssh binary. It:

Detects when you connect vscode to your remote slurm cluster,
Checks for an already running vscode job using the jobname,
Connects to existing job if it exists, creates a new job if not,
Proxyjumps to the compute node that your job was assigned,
Starts a little "watcher" process that watches for when you disconnect vscode/close the window on your local machine,
Connects!
Then when you close your local vscode window, if you don't reconnect within a timeout your vscode slurm job is cancelled for you! 🥳 🎉

Finally this script is where I want it to be! I even battled through the Powershell to make a Windows version. Next I just need to make it into an extension that probes your cluster's slurm config and generates the appropriate cluster-specific ssh configs for different slurm resource combos on-the-fly in a GUI :-D

Hi all, feel free to ask me to be quiet about this in this thread, but I've managed to get my vscode slurm ssh wrapper script working really well now!

It is a wrapper around ssh that you point vscode to instead of your ssh binary. It:

Detects when you connect vscode to your remote slurm cluster,

Checks for an already running vscode job using the jobname,

Connects to existing job if it exists, creates a new job if not,

Proxyjumps to the compute node that your job was assigned,

Starts a little "watcher" process that watches for when you disconnect vscode/close the window on your local machine,

Connects!

Then when you close your local vscode window, if you don't reconnect within a timeout your vscode slurm job is cancelled for you! 🥳 🎉

Finally this script is where I want it to be! I even battled through the Powershell to make a Windows version. Next I just need to make it into an extension that probes your cluster's slurm config and generates the appropriate cluster-specific ssh configs for different slurm resource combos on-the-fly in a GUI :-D

I tried to run this script on a windows 10 laptop that connects via WSL2 to a Linux server and I just get a "Could not establish connection to "server_name": spawn UNKNOWN. The "Installation" was as described in the readme with the appropriate path changes in

Remote-SSH: Settings -> Remote.SSH: Path

and the changes to the ssh config. Using the.sh and the .ps1 scripts does not work, the error remains the same. I did not really work with a cluster before, so I do not really know where to even start to resolve these issues.

Eugene Teoh (@eugeneteoh)

Eugene Teoh (@eugeneteoh) We have implemented a hackish solution to get the SLURM_ variables to the VS code session by storing them into a file that is sourced during login. To do this properly we have added a trap that will clean up the environment variables file on job termination.

#!/bin/bash
#SBATCH --time=01:00:00
#SBATCH --job-name="code-tunnel"
#SBATCH --signal=B:TERM@60 # tells the controller
                           # to send SIGTERM to the job 60 secs
                           # before its time ends to give it a
                           # chance for better cleanup.

cleanup() {
    echo "Caught signal - removing SLURM env file"
    rm -f ~/.code-tunnel-env.bash
}

# Trap the timeout signal (SIGTERM) and call the cleanup function
trap 'cleanup' SIGTERM

# store SLURM variables to file
env | awk -F= '$1~/^SLURM_/{print "export "$0}' > ~/.code-tunnel-env.bash

/usr/sbin/sshd -D -p 2222 -f /dev/null -h ${HOME}/.ssh/id_ecdsa &
wait

Then have users add the following line to their .bashrc file:

# source slurm environment if we're connecting through code-tunnel
[ -f ~/.code-tunnel-env.bash ] && source ~/.code-tunnel-env.bash

Would you mind expanding on the full procedure. I'm not sure I understand.

So the SBATCH part reserves resources for the vs-code server to run on and saves some string that helps identify the node we got that we later want the vs-code server to start running on.

What do we do with the part?

Host hpc-job
    ProxyCommand ssh hpc "nc \$(squeue --me --name=tunnel --states=R -h -O NodeList) 2222"
    StrictHostKeyChecking no

What does something like that mean? Exactly this or should the real host be in there somewhere. Is ProxyCommand literally ProxyCommand or should it be something set elsewhere? And is this for a ./ssh/config file on the remote cluster or locally (on the device from which VS-code is originally run before any ssh-ing)?

Is a change in VS-code required (mentioned in the thread but not specifically in this solution)?

Hi all, feel free to ask me to be quiet about this in this thread, but I've managed to get my vscode slurm ssh wrapper script working really well now!
It is a wrapper around ssh that you point vscode to instead of your ssh binary. It:

Detects when you connect vscode to your remote slurm cluster,

Checks for an already running vscode job using the jobname,

Connects to existing job if it exists, creates a new job if not,

Proxyjumps to the compute node that your job was assigned,

Starts a little "watcher" process that watches for when you disconnect vscode/close the window on your local machine,

Connects!

Then when you close your local vscode window, if you don't reconnect within a timeout your vscode slurm job is cancelled for you! 🥳 🎉

Finally this script is where I want it to be! I even battled through the Powershell to make a Windows version. Next I just need to make it into an extension that probes your cluster's slurm config and generates the appropriate cluster-specific ssh configs for different slurm resource combos on-the-fly in a GUI :-D

I tried to run this script on a windows 10 laptop that connects via WSL2 to a Linux server and I just get a "Could not establish connection to "server_name": spawn UNKNOWN. The "Installation" was as described in the readme with the appropriate path changes in

Remote-SSH: Settings -> Remote.SSH: Path

and the changes to the ssh config. Using the.sh and the .ps1 scripts does not work, the error remains the same. I did not really work with a cluster before, so I do not really know where to even start to resolve these issues.

Hi, sorry I haven't replied to this already, and feel free to open an issue in my repo to not clog up the thread. I did the windows version as a little extra/bonus (as not many people need it where I work) so it isn't as mature/robust as the linux solution. I have however pushed some fixes + some guidance for windows. It should be: edit ssh.bat to point to the location of ssh_wrapper.ps1, change vscode to point to ssh.bat as the ssh binary path, change/check vscode settings (local server true, remote command true etc.), define slurm job resources in the remote command of a Host in your ssh_config (like in the readme), cross fingers and connect. Feel free to share ssh connection logs in an issue in my repo, just click "details" when connecting :-)

Eugene Teoh (@eugeneteoh)

Eugene Teoh (@eugeneteoh) We have implemented a hackish solution to get the SLURM_ variables to the VS code session by storing them into a file that is sourced during login. To do this properly we have added a trap that will clean up the environment variables file on job termination.

#!/bin/bash
#SBATCH --time=01:00:00
#SBATCH --job-name="code-tunnel"
#SBATCH --signal=B:TERM@60 # tells the controller
                           # to send SIGTERM to the job 60 secs
                           # before its time ends to give it a
                           # chance for better cleanup.

cleanup() {
    echo "Caught signal - removing SLURM env file"
    rm -f ~/.code-tunnel-env.bash
}

# Trap the timeout signal (SIGTERM) and call the cleanup function
trap 'cleanup' SIGTERM

# store SLURM variables to file
env | awk -F= '$1~/^SLURM_/{print "export "$0}' > ~/.code-tunnel-env.bash

/usr/sbin/sshd -D -p 2222 -f /dev/null -h ${HOME}/.ssh/id_ecdsa &
wait

Then have users add the following line to their .bashrc file:

# source slurm environment if we're connecting through code-tunnel
[ -f ~/.code-tunnel-env.bash ] && source ~/.code-tunnel-env.bash

This is great. Made me wonder whether this is possible and it seemingly is? I haven't tested it yet.

Hi all, feel free to ask me to be quiet about this in this thread, but I've managed to get my vscode slurm ssh wrapper script working really well now!

It is a wrapper around ssh that you point vscode to instead of your ssh binary. It:

Detects when you connect vscode to your remote slurm cluster,

Checks for an already running vscode job using the jobname,

Connects to existing job if it exists, creates a new job if not,

Proxyjumps to the compute node that your job was assigned,

Starts a little "watcher" process that watches for when you disconnect vscode/close the window on your local machine,

Connects!

Then when you close your local vscode window, if you don't reconnect within a timeout your vscode slurm job is cancelled for you! 🥳 🎉

Finally this script is where I want it to be! I even battled through the Powershell to make a Windows version. Next I just need to make it into an extension that probes your cluster's slurm config and generates the appropriate cluster-specific ssh configs for different slurm resource combos on-the-fly in a GUI :-D

On some HPC clusters, a password is needed to be able to proxyjump to the compute nodes. Is there not a way around this? Like somehow running the vscode server upon allocation?

On some HPC clusters, a password is needed to be able to proxyjump to the compute nodes. Is there not a way around this? Like somehow running the vscode server upon allocation?

I'd hope it pops up a little window or you can type the password into the terminal? Without agent forwarding I have to type my password a bunch of times to get it to connect to the compute node.

The admins of the HPC i use have recently disabled SSH access from the frontend to allocated slurm nodes and now requires users to connect to running jobs with srun --jobid $job_id --overlap --pty bash thus disabling the rather well working proxyjump.

i have made a script to get a jobid and connect to it, and it works through a normal cli (ssh -t). Is there still no decent way to connect to a node through a frontend with some intermediary command?

Is there still no decent way to connect to a node through a frontend with some intermediary command?

Sadly not. The way VSCode communicates to the compute nodes strictly requires full ssh capabilities, using a tty proxy as srun --pty cannot work. I banged my head over and over on it.

You may ask your admins to enable pam_slurm_adopt.so so that you can still ssh into nodes, but only if you have a job running in them. This is accepted to many HPC centers nowadays, if they are running slurm as it appears given you mention srun

You may ask your admins to enable pam_slurm_adopt.so so that you can still ssh into nodes, but only if you have a job running in them. This is accepted to many HPC centers nowadays, if they are running slurm as it appears given you mention srun

i was given the reasoning that sshing into the nodes was not respecting resource allocations from slurm, thus allowing you to steal resources from other users if you were sharing the node. so i guess im forced to use the remote tunnels instead.

i was given the reasoning that sshing into the nodes was not respecting resource allocations from slurm, thus allowing you to steal resources from other users if you were sharing the node.

AFAIK, that's incorrect: https://slurm.schedmd.com/pam_slurm_adopt.html

Hi there,

I'm a HPC sysadmin here at Contextual AI.

Some of our users were following these instructions and sourced exported the slurm variables in their .bashrc .
It caused a ton of problem launching jobs on different nodes because these variables were being picked up in some of their job scripts.

Ended up coming to this solution:

if [[ -v SSH_AUTH_SOCK ]];
then

        if [[ ${SSH_AUTH_SOCK} =~ vscode ]];

        then

                [ -f ~/.code-tunnel-env.bash ] && source ~/.code-tunnel-env.bash

        fi


fi

Since VScode likes to use this variable and socket file, it allows for the variable to be used for VScode and shields it from other job launches.

_RC_ (@romxero)

_RC_ (@romxero) - Unfortunatelly it seems VS code is not consistent in setting this variable as in my case it's unset

Marten Reehorst (@martenreeh)

@eugeneteoh We have implemented a hackish solution to get the SLURM_ variables to the VS code session by storing them into a file that is sourced during login. To do this properly we have added a trap that will clean up the environment variables file on job termination.
#!/bin/bash
#SBATCH --time=01:00:00
#SBATCH --job-name="code-tunnel"
#SBATCH --signal=B:TERM@60 # tells the controller
# to send SIGTERM to the job 60 secs
# before its time ends to give it a
# chance for better cleanup.

cleanup() {
echo "Caught signal - removing SLURM env file"
rm -f ~/.code-tunnel-env.bash
}

Trap the timeout signal (SIGTERM) and call the cleanup function

trap 'cleanup' SIGTERM

store SLURM variables to file

env | awk -F= '$1~/^SLURM_/{print "export "$0}' > ~/.code-tunnel-env.bash

/usr/sbin/sshd -D -p 2222 -f /dev/null -h ${HOME}/.ssh/id_ecdsa &
wait

Then have users add the following line to their .bashrc file:

source slurm environment if we're connecting through code-tunnel

[ -f ~/.code-tunnel-env.bash ] && source ~/.code-tunnel-env.bash

Would you mind expanding on the full procedure. I'm not sure I understand.

So the SBATCH part reserves resources for the vs-code server to run on and saves some string that helps identify the node we got that we later want the vs-code server to start running on.

What do we do with the part?
Host hpc-job
    ProxyCommand ssh hpc "nc \$(squeue --me --name=tunnel --states=R -h -O NodeList) 2222"
    StrictHostKeyChecking no
What does something like that mean? Exactly this or should the real host be in there somewhere. Is ProxyCommand literally ProxyCommand or should it be something set elsewhere? And is this for a ./ssh/config file on the remote cluster or locally (on the device from which VS-code is originally run before any ssh-ing)?

Is a change in VS-code required (mentioned in the thread but not specifically in this solution)?

Marten Reehorst (@martenreeh) : did you get this to work? I don't get how the code-tunnel-env.bash is used afterwards.

Hi all,

All of the methods our team has come up with have been a bit overly complicated for users, so our latest solution is now this:

A vscode extension for users. Source here.

Hopefully users will now find it easier to debug code etc. without having to understand ssh configs and remote commands etc.

Navigation Menu

Repository navigation

Cluster workflow feature to allow shell commands or script to run before remote server setup (e.g. slurm) (wrap install script) #1722

107 remaining items

Trap the timeout signal (SIGTERM) and call the cleanup function

store SLURM variables to file

source slurm environment if we're connecting through code-tunnel

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Cluster workflow feature to allow shell commands or script to run before remote server setup (e.g. slurm) (wrap install script) #1722

Description

Activity

wwarriner commented on Oct 24, 2019

roblourens commented on Oct 27, 2019

wwarriner commented on Oct 28, 2019

roblourens commented on Oct 28, 2019

wwarriner commented on Oct 28, 2019

nicocarbone commented on Nov 6, 2019

daferna commented on Nov 13, 2019

107 remaining items

xangma commented on Mar 27, 2024

NoCreativeIdeaForGoodUserName commented on Apr 22, 2024

martenreeh commented on Apr 22, 2024

xangma commented on May 2, 2024

xangma commented on Jul 29, 2024

mredenti commented on Aug 2, 2024

xangma commented on Aug 30, 2024

PeterKADam commented on Dec 11, 2024

davide-q commented on Dec 11, 2024

PeterKADam commented on Dec 11, 2024

davide-q commented on Dec 11, 2024

romxero commented on Feb 28, 2025

Domomod commented on Apr 18, 2025

ppaxisa commented on Sep 26, 2025

Trap the timeout signal (SIGTERM) and call the cleanup function

store SLURM variables to file

source slurm environment if we're connecting through code-tunnel

xangma commented on Jan 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions