Merge pull request CycleCloud#4 from CycleCloud/bugfix/shiran-lab3-updates

Updates, mostly to Lab 3

Author: jermth

Lab1/Tutorial.md

@@ -6,15 +6,17 @@ for orchestrating HPC clusters in Azure.
 Please send questions or comments to [the Azure CycleCloud PM team](mailto:askcyclecloud@microsoft.com).
 
 ## Goals
+
+In this lab you will learn how to:
+
 * Create a fully functional, configured Azure CycleCloud instance that can be
-  used for further labs, pilot deployments, etc. 
-* Learn how to configure and deploy a LAMMPS HPC cluster, one of the standard
+  used for further labs, pilot deployments, etc.
+* Configure and deploy a LAMMPS HPC cluster, one of the standard
   cluster types included with Azure CycleCloud.
 
 ## Pre-requisites
-* Standard lab
-  [prerequisites](https://github.com/CycleCloud/cyclecloud_tutorials/blob/master/README.md#prerequisites)
-  
+
+* Standard lab [prerequisites](/README.md#prerequisites)
 
 ## 1. Starting An Azure CycleCloud Server
 
@@ -31,7 +33,8 @@ As you follow the steps below, *please keep track of the following*:
 
 
 ### 1.1 Log into (Azure Cloud Shell)[https://shell.azure.com]
-```
+
+```sh
 Requesting a Cloud Shell.Succeeded.
 Connecting terminal...
 
@@ -43,44 +46,52 @@ ellen@Azure:~$
 ```
 
 ### 1.2 Figure out your username
-```
+
+```sh
 ellen@Azure:~$ whoami
 ellen
 ellen@Azure:~$
 ```
 
 ### 1.3 Generate an SSH keypair (if you do not already have one)
+
 * In Cloud Shell, run:
-```
-ellen@Azure:~$ ssh-keygen -f ~/.ssh/id_rsa  -N "" -b 4096
-Generating public/private rsa key pair.
-Your identification has been saved in /home/ellen/.ssh/id_rsa.
-Your public key has been saved in /home/ellen/.ssh/id_rsa.pub.
-The key fingerprint is:
-SHA256:L8DFLyfXbKQT5PZZFwTGFnAY4ODCtYFyhGhoHFTpbKM ellen@cc-c6733553-7b96c85fb8-hbmlw
-The key's randomart image is:
-+---[RSA 4096]----+
-|+o+.+..+ .oo==+o |
-|.= +.oo.=o .oo  .|
-|o o oo oo.+ o . .|
-|   = ... o B o . |
-|  o . o S * *    |
-| E     . * o     |
-|        . .      |
-|         .       |
-|                 |
-+----[SHA256]-----+
-ellen@Azure:~$
-```
+
+  ```sh
+  ellen@Azure:~$ ssh-keygen -f ~/.ssh/id_rsa  -N "" -b 4096
+  Generating public/private rsa key pair.
+  Your identification has been saved in /home/ellen/.ssh/id_rsa.
+  Your public key has been saved in /home/ellen/.ssh/id_rsa.pub.
+  The key fingerprint is:
+  SHA256:L8DFLyfXbKQT5PZZFwTGFnAY4ODCtYFyhGhoHFTpbKM ellen@cc-c6733553-7b96c85fb8-hbmlw
+  The key's randomart image is:
+  +---[RSA 4096]----+
+  |+o+.+..+ .oo==+o |
+  |.= +.oo.=o .oo  .|
+  |o o oo oo.+ o . .|
+  |   = ... o B o . |
+  |  o . o S * *    |
+  | E     . * o     |
+  |        . .      |
+  |         .       |
+  |                 |
+  +----[SHA256]-----+
+  ellen@Azure:~$
+  ```
+
 ### 1.4 Retrieve the SSH-pubkey
-```
-ellen@Azure:~$ cat ~/.ssh/id_rsa.pub
-ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCwIvmC4K/0BUwOBqCsPxw5Ht8qWyDkorrU+gc6cJbohREoMFZkMlGEe2XqIyYTpHAu0ISicZEJ4MoWExPFrrZRqoYAHPrHyNnie9tVR5UMkqzNhs31qEWLtfEBOrcJIsPNdFuvnAqiQnhMQut3Jtamjc3XnMU8kJ3yL/+xIU4vKkQ8XIey+GGowR69oJI37mYKr9jT9dejB4gP2l6JyrVehnOG6QXRtg/gzFgyX08u8wKhuohNIPLlf2VzIXQml69P9PcD3muePIxi/JsJ6hb6czMCqhyHSFA42XpUpeWTml41HuBO9R5Bcsb3Q3j4MTKQOjtssz9Dx3pwtZ+tn9mg8TLMsk9d3Ip8FVXbe9ABleutJLIYGIUcZ3GlMdnRP62Wdyzrh0VEsoCfkHjUq2qFo8Hd1j0bkR1coSr2vFZXz6my+92a8nX7dJMPH5y+DG+ZuZchBXrwy8xVSNccnqRRn1A5xKdxY5SusbUQEirYPS7oR64CH4QeH6d5iQ2p2Z2cVWYbz/DjJNoCF0Cbzp9w1KprpErlrtd1epIGQTUDgx4YhChyrdXQiYCJBJ1jvhJcWfKj6rHz94eTEr6S5W4etP/IuACqKTpmAxQqSdU7NdHZVPH8c6w89JX3DAYRjg0PKVm56Ib6C+kct8M6/NQQeyhi5DM0SGW+T7tBjDxsUQ== ellen@cc-c6733553-7b96c85fb8-hbmlw
-ellen@Azure:~$
-```
-### 1.5 Create a service principal
+
+  ```sh
+  ellen@Azure:~$ cat ~/.ssh/id_rsa.pub
+  ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCwIvmC4K/0BUwOBqCsPxw5Ht8qWyDkorrU+gc6cJbohREoMFZkMlGEe2XqIyYTpHAu0ISicZEJ4MoWExPFrrZRqoYAHPrHyNnie9tVR5UMkqzNhs31qEWLtfEBOrcJIsPNdFuvnAqiQnhMQut3Jtamjc3XnMU8kJ3yL/+xIU4vKkQ8XIey+GGowR69oJI37mYKr9jT9dejB4gP2l6JyrVehnOG6QXRtg/gzFgyX08u8wKhuohNIPLlf2VzIXQml69P9PcD3muePIxi/JsJ6hb6czMCqhyHSFA42XpUpeWTml41HuBO9R5Bcsb3Q3j4MTKQOjtssz9Dx3pwtZ+tn9mg8TLMsk9d3Ip8FVXbe9ABleutJLIYGIUcZ3GlMdnRP62Wdyzrh0VEsoCfkHjUq2qFo8Hd1j0bkR1coSr2vFZXz6my+92a8nX7dJMPH5y+DG+ZuZchBXrwy8xVSNccnqRRn1A5xKdxY5SusbUQEirYPS7oR64CH4QeH6d5iQ2p2Z2cVWYbz/DjJNoCF0Cbzp9w1KprpErlrtd1epIGQTUDgx4YhChyrdXQiYCJBJ1jvhJcWfKj6rHz94eTEr6S5W4etP/IuACqKTpmAxQqSdU7NdHZVPH8c6w89JX3DAYRjg0PKVm56Ib6C+kct8M6/NQQeyhi5DM0SGW+T7tBjDxsUQ== ellen@cc-c6733553-7b96c85fb8-hbmlw
+  ellen@Azure:~$
+  ```
+
+### <a name="1.5"></a>1.5 Create a service principal
+
 Substitute a custom value for `cyclecloudlabs` as the name of your service principal. The name must be unique across all of Azure.
-```
+
+```sh
 ellen@Azure:~$ az ad sp create-for-rbac --name cyclecloudlabs
 {
     "appId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
@@ -90,7 +101,8 @@ ellen@Azure:~$ az ad sp create-for-rbac --name cyclecloudlabs
     "tenant": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
 }
 ```
-_Save this output somewhere. You will need it in the section below as well as in future tutorials in this lab._
+
+_Save this output somewhere. You will need it in the section below as well as later in this lab and in other labs._
 
 ### 1.6 Deploy Azure CycleCloud
 [![Deploy to
@@ -133,10 +145,10 @@ ellen@Azure:~$
   like and then click "Next".![First Login](images/cc-first-login.png)
 * Accept the End-User License Agreement and click "Next".
 * Create an admin user:
-    - For the User ID, use the same `username` used above in step 1.5. Remember 
-      that this is also the username of your Cloud Shell session.
-    - Enter a name for the user.
-    - Enter and confirm a new password that meets the minimum requirements.
+  * For the User ID, use the same `username` used above in step 1.5. Remember 
+    that this is also the username of your Cloud Shell session.
+  * Enter a name for the user.
+  * Enter and confirm a new password that meets the minimum requirements.
 
   ![Create Account](images/cc-create-account.png)
 
@@ -165,7 +177,7 @@ LAMMPS as a solver.
 * Click the "Save" button on the bottom to create this cluster.
 * Click the "Start" button to provision the cluster resources in Azure.
   ![CC Cluster Prepared](images/cc-cluster-prepared.png)
-  - Starting up the cluster for the first time takes about 10 minutess. By
+  * Starting up the cluster for the first time takes about 10 minutess. By
     default, only the master (or "head") node of the cluster is started. Azure
     CycleCloud provisions all the necessary network and storage resources needed
     by the master node, and also sets up the scheduling environment in the
@@ -174,7 +186,7 @@ LAMMPS as a solver.
   for the green status bar before proceeding to the next step.
   ![CC Cluster Ready](images/cc-cluster-ready.png)
 
-### 2.2 Connecting to the master node and submitting a LAMMPS job
+### <a name="2.2"></a> 2.2 Connecting to the master node and submitting a LAMMPS job
 
 The SSH public key you specified as part of the deployment is stored in the Azure CycleCloud application server and pushed into each cluster that you create. As a result, you can use your SSH private key to log into the
 master node.

Lab2/Tutorial.md

@@ -9,24 +9,26 @@ concepts such as projects and cluster template management.
 Please send questions or comments to [the Azure CycleCloud PM team](mailto:askcyclecloud@microsoft.com).
 
 ## Goals
-By the end of this lab, we will cover:
-* Installing and configuring the Azure CycleCloud `cyclecloud` CLI tool
-* Creating a new CycleCloud project, a way of configuring and customizing a
+
+In this lab you will learn how to:
+
+* Install and configure the Azure CycleCloud `cyclecloud` CLI tool
+* Create a new CycleCloud project, a way of configuring and customizing a
   CycleCloud cluster
-* Modifying a cluster template to add storage to the cluster's NFS server
-* Importing the modified cluster template into CycleCloud in order to add a new
-  cluster type
-* Starting the new cluster type, and verifying that the persistent storage has been
-  added to the file system
+* Modify a cluster template to add storage to the cluster's NFS server
+* Import cluster templates into CycleCloud in order to add new cluster type
+* Create a new cluster from a custom cluster type, start it, and verify that the persistent storage has been added to the file system
 
 ## Pre-requisites
+
 * Standard lab
   [prerequisites](https://github.com/CycleCloud/cyclecloud_tutorials/blob/master/README.md#prerequisites)
 
 ## 3. Modifying a cluster template
+
 Azure CycleCloud's cluster types are great for standard use cases. But
 occassionally users find themselves needing to customize the clusters for more
-advanced or differently configured deployments. 
+advanced or differently configured deployments.
 
 A common customization is adding managed disks to a VM in a
 compute cluster. By default, in most Azure CycleCloud clusters the master nodes
@@ -45,10 +47,11 @@ environment. Once again, we will use the [Azure Cloud Shell](https://shell.azure
 as our shell environment.
 
 ### 3.1 Installing and setting up the Azure CycleCloud CLI
+
 * If you are still logged onto the LAMMPS master node from the [previous lab](/Lab1/Tutorial.md),
   return to Cloud Shell by running the `exit` command.
 
-  ```
+  ```sh
   [ellen@ip-0A000404 ~]$ exit
   logout
   Connection to 40.114.123.148 closed.
@@ -57,7 +60,8 @@ as our shell environment.
 
 * Download the CycleCloud command line installer into the Cloud Shell
   environment by running the following command from the prompt:
-  ```
+
+  ```sh
   ellen@Azure:~$ wget https://cyclecloudarm.blob.core.windows.net/cyclecloudrelease/7.5.0/cyclecloud-cli.zip
   --2018-08-02 21:48:30--  https://cyclecloudarm.blob.core.windows.net/cyclecloudrelease/7.5.0/cyclecloud-cli.zip
   Resolving cyclecloudarm.blob.core.windows.net   (cyclecloudarm.blob.core.windows.net)... 52.239.154.132
@@ -71,8 +75,10 @@ as our shell environment.
   2018-08-02 21:48:31 (112 MB/s) - ‘cyclecloud-cli.zip’ saved [4546572/4546572]
   ellen@Azure:~$
   ```
+
 * Unzip the file:
-  ```
+
+  ```sh
   ellen@Azure:~$ unzip cyclecloud-cli.zip 
   Archive:  cyclecloud-cli.zip
      creating: cyclecloud-cli-installer/
@@ -93,18 +99,20 @@ as our shell environment.
     inflating: cyclecloud-cli-installer/packages/pogo-sdist.tar.gz
   ellen@Azure:~$
   ```
-  * Change into the unzipped install directory, and run the install script  
-  ```
+
+* Change into the unzipped install directory, and run the install script  
+
+  ```sh
   ellen@Azure:~$ cd cyclecloud-cli-installer
   ellen@Azure:~/cyclecloud-cli-installer$ ./install.sh
   cyclecloud and pogo commands have been installed to /home/ellen/bin
   ellen@Azure:~/cyclecloud-cli-installer$
   ```
 
-  - If you receive the error `'/home/ellen/bin' not found in your PATH
+  * If you receive the error `'/home/ellen/bin' not found in your PATH
   environment variable. Make sure to update it`, you can fix it as follows:
 
-    ```
+    ```sh
     ellen@Azure:~$ echo $PATH
     /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin:/opt/mssql-tools/bin
     ellen@Azure:~$ export PATH=~/bin:$PATH
@@ -116,15 +124,15 @@ as our shell environment.
 
   The Azure CycleCloud CLI communicates with the server over a REST API. To use
   it you first have to initialize it using the `initialize` command.
-    - The CycleServer URL is the FQDN of your application
+  * The CycleServer URL is the FQDN of your application
       server set up in [Lab 1 Section 1.7](/Lab1/Tutorial.md#1.7).
-    - The installed Azure CycleCloud server uses either a Let's Encrypt SSL
+  * The installed Azure CycleCloud server uses either a Let's Encrypt SSL
       certificate or a self-signed certificate. Type `yes` when asked if you allow an
       untrusted certificate.
-    - The CycleServer username is the one use to log into the CycleCloud web portal.
-    - The CycleServer password is the password for the user above.
+  * The CycleServer username is the one use to log into the CycleCloud web portal.
+  * The CycleServer password is the password for the user above.
 
-  ```
+  ```sh
   ellen@Azure:~$ cyclecloud initialize
   CycleServer URL: [http://localhost:8080] https://{FQDN}
   Detected untrusted certificate.  Allow?: [no] yes
@@ -145,7 +153,8 @@ as our shell environment.
 
 * Verify that the CycleCloud CLI works using the `show_cluster` command. The command
   should display the LAMMPS cluster you created in [Lab 1](/Lab1/Tutorial.md#2.1).
-  ```
+
+  ```sh
   ellen@Azure:~$ cyclecloud show_cluster
   --------------------
   LammpsLabs : started
@@ -155,9 +164,10 @@ as our shell environment.
       master: Started e6e008a1259743f8406967a023633a6a 40.114.123.148 (10.0.4.4)
   Total nodes: 1
   ellen@Azure:~$
-  ``` 
+  ```
+
+### <a name="3.2"></a> 3.2 Creating a new CycleCloud Project
 
-### 3.2 Creating a new CycleCloud Project
 Azure CycleCloud clusters are defined using text files. To take a look at one of
 these, use the CycleCloud CLI to create a new project, and generate a template
 from it.
@@ -166,13 +176,13 @@ from it.
 
   Create a parent directory for cyclecloud
   projects, then create a new project with the `cyclecloud project init`
-  command. 
+  command.
 
-    - In the example below, the project is named `azurecyclecloud_labs`. 
-    - When asked for the `Default Locker`, specify `azure-storage`. This was
+  * In the example below, the project is named `azurecyclecloud_labs`.
+  * When asked for the `Default Locker`, specify `azure-storage`. This was
       created for you during deployment.
 
-  ```
+  ```sh
   ellen@Azure:~$ mkdir ~/cyclecloud_projects/
   ellen@Azure:~$ cd ~/cyclecloud_projects/
   ellen@Azure:~/cyclecloud_projects$ cyclecloud project init azurecyclecloud_labs
@@ -192,29 +202,36 @@ from it.
   to modify the file but you can use another editor (such as `vi` or `emacs`).
 
   Add the line `type = application` into `project.ini` and save the changes.
-  ```
+
+  ```sh
   ellen@Azure:~/cyclecloud_projects$ cd ./azurecyclecloud_labs
   ellen@Azure:~/cyclecloud_projects/azurecyclecloud_labs$ code .
   ```
+
   ![Edit Project File](images/cloudshell-editor-project-ini.png)
   
 ### 3.3 Generate a new cluster template file
+
 * Run the following command to create a new cluster template. The last argument
   is the output file that will contain our new template.
 
-  ```
+  ```sh
   ellen@Azure:~/cyclecloud_projects/azurecyclecloud_labs$ cyclecloud project generate_template templates/pbs_extended_nfs.template.txt
   Cluster template written to templates/pbs_extended_nfs.template.txt
   ellen@Azure:~/cyclecloud_projects/azurecyclecloud_labs$
   ```
 
 ### 3.4 Add volumes to the NFS server in the template
+
 * Open the generated template file in an editor:
-  ```
+
+  ```sh
   ellen@Azure:~/cyclecloud_projects/azurecyclecloud_labs$ code templates/pbs_extended_nfs.template.txt
   ```
+
 * After line 44, add the following blocks:
-  ```INI
+
+  ```ini
           # Add 2 premium disks in a RAID 0 configuration to the NFS export
           [[[volume nfs-1]]]
           Size = 512
@@ -233,6 +250,7 @@ from it.
           fs_type = ext4
           raid_level = 0
   ```
+
 * Save the changes. The template file should now look like this: ![Edit Cluster Template](images/edit-cluster-template.png)
 
   These 15 lines express that two premium disks (`SSD = True`) of 512GB each
@@ -247,8 +265,10 @@ from it.
   [More information about customizing volumes and mounts in a CycleCloud cluster.](https://docs.microsoft.com/en-us/azure/cyclecloud/attach-storage)
 
 ### 3.5 Import the new cluster template
+
 * Using the CycleCloud CLI, import the template into the application server:
-  ```
+
+  ```sh
   ellen@Azure:~/cyclecloud_projects/azurecyclecloud_labs$ cyclecloud import_template -f templates/pbs_extended_nfs.template.txt
   Importing default template in templates/pbs_extended_nfs.template.txt....
   ---------------------------------
@@ -260,10 +280,12 @@ from it.
   Total nodes: 1
   ellen@Azure:~/cyclecloud_projects/azurecyclecloud_labs$
   ```
+
 * You should now see a new cluster type in the Azure CycleCloud UI:
   ![New Cluster Type](images/new-application-cluster.png)
 
 ### 3.6 Start the cluster
+
 * Follow the procedure in [Section 2 from Lab 1](/Lab1/Tutorial.md#2.1) to start
   a new cluster based on this new cluster type. Note that you must select a
   "Master VM Type" that supports attached premium storage, such as
@@ -272,7 +294,7 @@ from it.
 * After the cluster is started, log into the master node and verify that
   `/mnt/exports` is a 1TB volume:
 
-  ```
+  ```sh
   [ellen@ip-0A000405 ~]$ df -H /mnt/exports
   Filesystem                         Size  Used Avail Use% Mounted on
   /dev/mapper/vg_cyclecloud_nfs-lv0  1.1T   80M  1.1T   1% /mnt/exports