[Storage] Refactor storage and fix data transfer service

[Michaelvll]

Adopt changes for storage from #1152.

Tested:

• tests/run_smoke_tests.sh TestStorageWithCredentials
• The following command

gsutil mb gs://sky-imagenet-bucket-gcp
python - <<EOF
from sky.data import data_transfer
data_transfer.s3_to_gcs('sky-imagenet-bucket', 'sky-imagenet-bucket-gcp') 
EOF

[romilbhardwaj]

Thanks for fixing this @Michaelvll!

[romilbhardwaj]

Curious - are there any benefits other than readability to calling transferJobs().create() followed by transferJobs().run() instead of setting the schedule field to current time and only calling transferJobs().create() like we had before?

(Just to be clear, I prefer the former as we have now, but curious if there's some other reason to do it this way)

[Michaelvll]

The main reason for using the current interactive way to start the job is that we can get the name of the operation for the run().execute() (this is different from the name of the submitted TransferJob, in the response). With the name, we don't have to list all the running transfering jobs and find out the correct operation name that is scheduled by the cloud in L98 below. Also, since we will only run the TransferJob once and in a blocking manner, I feel like having a schedule field in the specification can be a bit misleading.

[romilbhardwaj]

lgtm! Thanks for fixing this!