[BUG]: Need a way to set Julia bind interface

[ViRb3]

What happened?

The built-in Slurm controller, when enabled, will bind on the first interface it finds when it tries to establish a connection between the nodes. If the nodes happen to have a public internet access, this means that the nodes will try to talk to each other over the public internet, which is usually firewalled. One needs to set the julia --bind-to=10.8.0.0.1 parameter, but I couldn't see a way to do it.

I ended up with this interesting workaround:

cd /opt/pysr-env/julia_env/pyjuliapkg/install/bin
if [ ! -f julia_real ]; then
    mv julia julia_real
fi
cat << EOT > julia
#!/bin/bash
exec "$PWD/julia_real" --bind-to=${PRIVATE_IP} "\$@"
EOT
chmod +x julia
cd -

This might be a problem with other distributed controllers as well.

[MilesCranmer]

Thanks! Actually do you think you could do a test drive of this new slurm_native feature here? #794. This should be more in-line with how slurm ought to work. It uses https://github.com/JuliaParallel/SlurmClusterManager.jl as a backend rather than SlurmManager.

I think that PR should mostly be ready to merge, actually. I wanted to get some unittests set up via https://github.com/giovtorres/slurm-docker-cluster but couldn't really get it to work yet. So I'm happy to merge before testing is in place if it helps with user side testing.

[ViRb3]

I can definitely give it a test tomorrow. I have a multi-node cloud setup, so if you have any specific testing criteria, let me know.

[ViRb3]

Actually, I want to note down another related detail before I forget - looks like the heap_size_hint_in_bytes is being calculated based on one the current node's RAM, instead of the total RAM of all nodes, so it ends up allocation something like 6MB per process by default with procs=160, which causes issues. Had to override the param and set it to something more sane like 1GB.