Working with Google Cloud Platform

Uploading from HPC to GCP bucket

Combiz did this in Nov 2020 to send Biogen sequencing data. It took a while to figure it out after a couple of dead ends (i.e. can't use google cli tools on the HCP as they need a browser to authenticate, and can't connect from the instance -> hcp as it requires a VPN)

He came up with this solution:

• create a bucket
• create a gcp instance
• use gcsfuse to mount the bucket on the instance
• generate an ssh key on the HCP and share the public key with the instance
• use scp inside tmux on the HCP to copy the RDS data to the mounted bucket folder on the instance