Updated docs, added start_n_more

Author: yadudoc

aws/README.asc

@@ -1,7 +1,6 @@
 Swift cloud on Amazon Elastic Compute Cloud
 -------------------------------------------
 
-
 Sign up online
 ^^^^^^^^^^^^^^
 
@@ -28,8 +27,7 @@ Secret access key example: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY +
 6. Click Download Credentials, and store the keys (a .csv file) in a secure location.
 
 NOTE: Your secret key will no longer be available through the AWS Management Console;
-you will have the only copy. Keep it confidential in order to protect your account,
-and never email it.
+you will have the only copy. Keep it confidential in order to protect your account.
 
 Setup local-machine
 ^^^^^^^^^^^^^^^^^^^
@@ -41,36 +39,87 @@ git clone git@github.com:yadudoc/swift-on-cloud.git
 cd swift-on-cloud/aws
 -----
 
-
-* TODO
-
 Manage you Cloud resources
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-* TODO
+Update the file swift-on-cloud/aws/configs in the cloned repository from the previous step,
+with the following information:
+ *  AWS_CREDENTIALS_FILE : Path to the credentials file downloaded in step 6 of _sign_up_online
+ *  AWS_WORKER_COUNT : Use this to specify the number of worker instances required.
+ *  AWS_KEYPAIR_NAME : Name of the keypair to use. If this keypair does not exist a keypair of the same name will be generated.
+ *  AWS_KEYPAIR_FILE : Path to the <keypair>.pem. If a keypair file of the specified name does not exist, a new one will be generated.
+ *  AWS_USERNAME     : The username used for login to the cloud-instances. Set to the default "ec2-user" for Amazon Linux AMI.
+ *  AWS_REGION       : Default=us-west-2 | Do NOT change
+ *  SECURITY_GROUP   : Default=swift_security_group1
+ *  HEADNODE_IMAGE, WORKER_IMAGE : These are images used to boot up the headnode and workers.
+ *  HEADNODE_MACHINE_TYPE, WORKER_MACHINE_TYPE : VM types for workers and the headnode. Choose between :
+    - t2.<micro/small/medium>, m3.<medium,large,xlarge,2xlarge> :For general purpose computing
+    - c3.<,x,2x,4x,8x>large             : Compute Optimised
+    - g2.2xlarge                        : GPU Instances
+    - r3.<,x,2x,4x,8x>large             : Memory Optimised
+    - i2.<x,2x,4x,8x>large, hs1.8xlarge : Storage Optimised
+
+NOTE: Get details of instances and pricing http://aws.amazon.com/ec2/pricing[here]. The pricing varies
+between geographical locations so make sure you check for US West Oregon, which is the default.
+
+NOTE: GCE_WORKER_COUNT directly affects the cost. If you require more than 22 nodes including the
+headnode, file a request to increase your resource quotas.
+
+NOTE: Read more about AWS regions and availability zones http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html[here]
+
+WARNING:  Do *NOT* change the images for the worker and headnode to point at images which have not
+been explicitly setup with swift.
 
 Start your cloud setup!
 ^^^^^^^^^^^^^^^^^^^^^^^
 
-* TODO
-
-Run swift from your local machine
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Once you finish editing the configs file with your preferences, start the cloud instances
+by sourcing the setup script.
 
-* TODO
+The setup script will setup firewall rules, copy over the required images and start a headnode
+and the requested number of worker instances.
 
+[source, bash]
+-----
+# Must source the setup script.
+source setup.sh
+-----
 
 Run the tutorial on the cloud
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-* TODO
+Once your cloud resources have been configured and setup, you can run the swift-cloud-tutorial
+directly from the cloud. The cloud resources created include a headnode, to which you would
+connect to, and the several nodes in worker roles which would do computations in parallel.
 
+To run the tutorial, first connect to the headnode:
 
-Miscellanious operations supported:
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-* TODO
+[source, bash]
+-----
+# Connect to the Headnode
+connect headnode
+# This will have you logged in to the headnode on the cloud
+-----
+
+
+Run swift from your local machine
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Now, once you've configured and started you cloud resources we can move on to trying the
+swift-cloud-tutorial. To run the cloud tutorial from your local machine :
 
+[source,bash]
+-----
+cd swift-on-cloud/swift-cloud-tutorial
+source setup.sh
+-----
+
+NOTE: You must source the setup.sh script.
+
+
+Miscellanious operations supported:
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 [source, bash]
 -----
@@ -118,4 +167,6 @@ start_n_more <Number of nodes>
 source setup.sh
 -----
 
-Note: Creating an Image https://developers.google.com/compute/docs/images#creatingimage[Link]
+Note: To create your own customised images, you can connect to the swift instances (headnodes/workers) and
+modify them. Once done you can create your own images. Documentation on Amazon Machine Images is available
+http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html[here]

aws/aws.py

@@ -131,7 +131,6 @@ def start_worker(driver, configs, worker_names):
     # Setup userdata
     userdata   = configurator.getstring("worker")
     userdata   = userdata.replace("SET_HEADNODE_IP", headnode.public_ips[0])
-    logging.info("Headnode connection url : ",headnode.public_ips[0])
     logging.debug("Worker userdata : %s", userdata)
 
     list_nodes = []

aws/configs

@@ -1,27 +1,22 @@
 AWS_CREDENTIALS_FILE=~/.ssh/boto-test-credentials.csv
 
-#AWS_KEYPAIR_NAME=boto-test-pair
-#AWS_KEYPAIR_FILE=~/.ssh/boto-test-pair.pem
+# Set the number of worker nodes required
+AWS_WORKER_COUNT=5
 
 # Update the following only if you want to use a specific keypair
 AWS_KEYPAIR_NAME=swift-test-pair
 AWS_KEYPAIR_FILE=~/.ssh/swift-test-pair.pem
 
-
 #ec2-user is the default user on Amazon linux
-#AWS_USERNAME=ec2-user
+AWS_USERNAME=ec2-user
 
 AWS_REGION=us-west-2
+SECURITY_GROUP=swift_security_group1
 
 # Amazon linux x86_64 with Swift and JDK
 HEADNODE_IMAGE=ami-f1e896c1
-
 HEADNODE_MACHINE_TYPE=t1.micro
 
-AWS_WORKER_COUNT=5
-
-SECURITY_GROUP=swift_security_group1
-
+# Amazon linux x86_64 with Swift and JDK
+WORKER_IMAGE=ami-f1e896c1
 WORKER_MACHINE_TYPE=t1.micro
-# Ami for workers
-WORKER_IMAGE=ami-f1e896c1

aws/setup.sh

@@ -61,13 +61,8 @@ start_n_more ()
 {
     ACTIVE=$(./aws.py list_resources | grep worker | wc -l)
     MORE=$1
-    for i in $(seq $(($ACTIVE+1)) 1 $(($ACTIVE+$MORE)) )
-    do
-        echo "Starting worker $i"
-        ./aws.py start_worker swift-worker-$i &> $LOG &
-    done
-    wait
-    ./aws.py list_resources
+    start_worker $(printf "swift-worker-%03d " $(seq $(($ACTIVE+1)) 1 $(($ACTIVE+$MORE)) ) )
+    list_resources
 }
 
 stop_headnode()