Skip to content

Latest commit

 

History

History
189 lines (135 loc) · 7.59 KB

README.md

File metadata and controls

189 lines (135 loc) · 7.59 KB

AutoDRIVE Simulator: Palmetto Cluster Deployments

This branch hosts resources for deployments of the AutoDRIVE Simulator on Palmetto 2 HPC cluster.

Submit Job to Palmetto 2 HPC Cluster

The overall project objective focuses on variability testing of the AEB functionality (SUT) for OpenCAV. Each test is stored in its own sub-folder, under the palmetto directory. In order to reproduce any one of these tests, you will need to upload the corresponding sub-folder to your Palmetto 2 home folder, update the relevant paths within each script, and then submit the corresponding job script.

sbatch autodrive_test.sh

Create conda Environment

All tests also rely on the autodrive conda environment being available to you on Palmetto 2. This environment can be recreated, as needed, from the file palmetto/environment.yml, using the following command:

conda env create -n autodrive -f environment.yml

Build AutoDRIVE Simulator Image

It is assumed that you have an installation of the required version of AutoDRIVE Simulator. Please refer to the instructions below to set up a containerized installation of AutoDRIVE Simulator by use of a sandboxed Singularity container. In order to perform the build process, you first need to procure a Docker image for AutoDRIVE Simulator. This image can either be built using the instructions provided in this same repository, under the docker branch, or the autodriveecosystem/autodrive_sim_opencav pre-built image available on Docker Hub may be pulled and used instead.

You may build a Singularity sandbox for it under your Palmetto 2 home folder using the following command:

cd ~
singularity build --sandbox autodrive_simulator/ docker://autodriveecosystem/autodrive_sim_opencav

Start Instance of the Sandbox

Once the image is built and written to the destination, start the instance of the sandbox using the following command:

singularity instance start --nv -B $HOME,$TMPDIR autodrive_simulator/ inst1

Run Instance of the Sandbox

Run the started instance of the sandbox using the following command (the writable flag allows us to make changes within the container should we need to save these changes as images):

singularity run --writable --nv -B $HOME,$TMPDIR instance://inst1

Setting up the Terminal

Install MobaXterm from here: https://mobaxterm.mobatek.net/

In sessions, create a new session and select SSH. image

In remote host, enter: slogin.palmetto.clemson.edu Make sure Specify username is checked and enter your username. In advanced SSH settings, make sure X11-Forwarding is enabled. Click OK to create the session Use this session for working with AutoDRIVE on Palmetto2.

Running the Simulator with an Interactive Job:

Run this command to start an interactive job with 2 k40 GPUs:

salloc --nodes=1 --tasks-per-node=4 --x11 --mem=16g --time=01:00:00 --gpus-per-node k40:2

Start and run the instance of the sandbox with these commands (same as mentioned above)

singularity instance start --nv -B $HOME,$TMPDIR autodrive_simulator/ inst1
singularity run --writable --nv -B $HOME,$TMPDIR instance://inst1

Run this command:

export DISPLAY="$(xauth list | grep "$SLURMD_NODENAME" | tail -n 1 | grep -oE :[0-9]+)"

Make sure you have GPU with ID 1 assigned (that’s why we’re asking for 2 GPUs):

echo ${SLURM_STEP_GPUS:-$SLURM_JOB_GPUS}

Go to autodrive/autodrive_simulator/home/ AutoDRIVE_Simulator/ Run the simulator with:

./AutoDRIVE\ Simulator.x86_64

Running the Tests:

Take a look at interactive sessions for palmetto2: https://ondemand.rcd.clemson.edu/pun/sys/dashboard/batch_connect/sessions

You can use Palmetto Desktop if you’d like to work with palmetto through a GUI (this will be required to see the webviewer results). You will still need to use MobaXterm to run the tests. There is also a Code Server (VSCode) that makes it easier to work with the file system. You can also use VSCode on your local machine for the same.

Ensure that you have this repository cloned in your folder on palmetto2. If this is done already, you should see a folder called AutoDRIVE-Simulator-HPC.

Headless (No-Graphics)

Go to AutoDRIVE-Simulator-HPC/palmetto/autodrive_test_no_graphics. Delete any .csv, .log or .out files present. These files are created on test execution, so you shouldn’t see them the first time you try to execute the test. In the same folder, open autodrive_test.sh and edit it so the directory paths have your username in them instead. image

In your MobaXterm SSH session, go to AutoDRIVE-Simulator-HPC/palmetto/autodrive_test_no_graphics Submit the batch job with:

sbatch autodrive_test.sh

Record-to-File

First go to autodrive_simulator/home/output and make sure that it’s empty. Delete any files present (this is where the recorded output of the previous execution will be stored). Go to AutoDRIVE-Simulator-HPC/palmetto/autodrive_test_record_to_file. Delete any .csv, .log or .out files present. In the same folder, open autodrive_test.sh and edit it so the directory paths have your username in them instead.

In your MobaXterm SSH session, go to AutoDRIVE-Simulator-HPC/palmetto/autodrive_test_record_to_file Submit the batch job with:

sbatch autodrive_test.sh

You can find the output files in autodrive_simulator/home/output. You can download them to your local machine by running this command (from your local machine).

scp <username>@hpcdtn01.rcd.clemson.edu:/home/<username>/<path to file> .

Alternatively, if you find the output file in the the same place in VSCode through the interactive session and right click on it, you can directly download it from there.

Live-Streaming

Go to AutoDRIVE-Simulator-HPC/palmetto/autodrive_test_webviewer. Delete any .csv, .log or .out files present. In the same folder, open autodrive_test.sh and edit it so the directory paths have your username in them instead. In your MobaXterm SSH session, go to AutoDRIVE-Simulator-HPC/palmetto/autodrive_test_webviewer Submit the batch job with: sbatch autodrive_test.sh To deploy the Webviewer, launch another interactive job and in the same folder, run:

bash deploy_webviewer.sh

image

In palmetto desktop, go to http://node0352.palmetto.clemson.edu:8000/webviewer/ (replace node0352 with whatever node your job is running on). Here, you will see the live feed of the test.

In your MobaXterm SSH session, go to AutoDRIVE-Simulator-HPC/palmetto/autodrive_test_record_to_file Submit the batch job with:

sbatch autodrive_test.sh

Results

Results

Some Useful Commands

Show jobs:

sacct -X

Show jobs that are running/pending:

sacct -X -s R,PD

Show status of running/pending jobs along with the reason for being held in queue:

squeue -u <username>

Details about the job (like scheduled start time etc.):

scontrol show job <job id>

Kill job (for a batch job, only enter the primary job id (without underscore)):

scancel <job ID>

Kill all jobs for your user

scancel -u <username>