README.md

Tips for HPC and VPS

You can import this repository as your private repository. Therefore, you can easily edit and copy commands.

Navigation

• basic linux
• basic HPC
• SSH for connecting to server
  • SSH without password (using key)
  • SSH config file
• singularity container
• useful scripts
• set up personal environment modules or Lmod

Common Issues

• Singularity: use :ro when otherwise running containers
• --nv: pass NVIDIA drivers to singularity containers
• do not use OpenOnDemand, use local forward and remote forward!

Tools

• Rclone
• OpenConnect (Not working due to authentication change)

Introducing mamba (Now conda use libmamba, so it doesn't matter.)

Highly recommend mamba instead of conda to manage R and Python packages.

It is quite handy in handling R packages (tidyverse, BiocManager, ...), whereas conda is prone to encounter conflicts that take forever to solve.

Quick Tips: alias

You can modify your ~/.bashrc, so you can run some commands easily.

Aliases are like custom shortcuts used to represent a command (or set of commands)

alias ls="ls --color=auto"
alias la="ls -A"
alias ll="ls -lh"
alias myjobs="squeue -u $USER"

next time you type ll, it equals to ls --color=auto -lh

VS Code on HPC

You can also use VS Code on HPC's compute node, see VS Code - NYU HPC. But the python extension has about 10,000 files, while the home quota is 30,000.

Install extensions in $SCRATCH

"remote.SSH.serverInstallPath": {
  "greene": "/scratch/zz999/"
}

Use Bash to get filename or extension

$ FILE=example.tar.gz

$ echo "${FILE%%.*}"
example

$ echo "${FILE%.*}"
example.tar

$ echo "${FILE#*.}"
tar.gz

$ echo "${FILE##*.}"
gz