hands-on-aws-2-sqs-pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hands-on AWS - 2 - SQS pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Installing dependencies\n",
    "**Here we install relevant dependencies to run serverless framework.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install awscli --upgrade --user\n",
    "!npm install -g serverless\n",
    "!echo -e '\\n\\033[1mStep finished\\033[0m'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting AWS environmental variables\n",
    "**Here we set up AWS environmental variables so that we will be able to deploy to our AWS account. Please replace it with provided credentials.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%env AWS_ACCESS_KEY_ID=<AWS_ACCESS_KEY_ID>\n",
    "%env AWS_SECRET_ACCESS_KEY=<AWS_SECRET_ACCESS_KEY>\n",
    "%env AWS_ACCOUNT_ID=<AWS_ACCOUNT_ID>"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Setting up identifier\n",
    "**Here we set up personal identifier. You will need to replace \"FirstName LastName\" with your first name and lastname.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "import hashlib\n",
    "import datetime\n",
    "res=hashlib.sha224(bytes(\"FirstName LastName\"+str(datetime.datetime.now().timestamp()),'utf-8')).hexdigest()[:5]\n",
    "%env HASH=$res\n",
    "%env AWS_DEFAULT_REGION=us-west-1"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## SQS+Lambda deployment\n",
    "\n",
    "**Let's take a look at the code which will run within lambda. The code will read the message and save it to S3 with message id as key and body as content.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!cat hands-on-aws-2-sqs-pipeline/index.py"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Now let's deploy the lambda with SQS queues. It may a minute or two to get deployed.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd hands-on-aws-2-sqs-pipeline;npm install\n",
    "!cd hands-on-aws-2-sqs-pipeline;serverless deploy --verbose > logs.txt\n",
    "!echo -e '\\n\\033[1mStep finished\\033[0m'"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Let's view the output of the deployment process. Serverless framework will return the queues which we created.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!cd hands-on-aws-2-sqs-pipeline;tail -15 logs.txt\n",
    "!echo -e '\\n\\033[1mStep finished\\033[0m'"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Sending message to SQS and looking at the logs\n",
    "\n",
    "**Let's send simple message to the SQS and check the it's read by the lambda.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws sqs send-message --queue-url $(cd hands-on-aws-2-sqs-pipeline;tail -15 logs.txt | grep \"ReadQueueUrl: \" | cut -f2- -d:) --message-body \"HelloWorld\"\n",
    "!echo -e '\\n\\033[1mStep finished\\033[0m'"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Now let's take a look at the latest logs for the function and check that it received the message.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!sleep 10\n",
    "!cd hands-on-aws-2-sqs-pipeline;serverless logs -f HelloWorld\n",
    "!echo -e '\\n\\033[1mStep finished\\033[0m'"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Let's check that lambda wrote the message to S3 by checking message id key.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!aws s3 cp --quiet s3://vhive-aws-tutorial/$(cd hands-on-aws-2-sqs-pipeline;serverless logs -f HelloWorld | grep -m 1 \"Message id:\" | cut -f2- -d:) /dev/stdout\n",
    "!echo -e '\\n\\033[1mStep finished\\033[0m'"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Now let's send a message which will result in error on lambda side.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!aws sqs send-message --queue-url $(cd hands-on-aws-2-sqs-pipeline;tail -15 logs.txt | grep \"ReadQueueUrl: \" | cut -f2- -d:) --message-body \"ValueError\"\n",
    "!echo -e '\\n\\033[1mStep finished\\033[0m'"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Let's check how this message will be read by lambda.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!sleep 10\n",
    "!cd hands-on-aws-2-sqs-pipeline;serverless logs -f HelloWorld\n",
    "!echo -e '\\n\\033[1mStep finished\\033[0m'"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "**Since the lambda failed the message will be resent to the dead letter queue.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "!sleep 10\n",
    "!aws sqs receive-message --queue-url $(cd hands-on-aws-2-sqs-pipeline;tail -15 logs.txt | grep \"DeadLetterQueueUrl: \" | cut -f2- -d:)\n",
    "!echo -e '\\n\\033[1mStep finished\\033[0m'"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Remove infrastructure\n",
    "\n",
    "**Now let's do the cleanup process and remove the infrastructure which we created.**"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "!cd hands-on-aws-2-sqs-pipeline;serverless remove > logs.txt\n",
    "!echo -e '\\n\\033[1mStep finished\\033[0m'"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}