Interview-Pairing

Problem

How do we ensure you will love working at CloudZero?

Concepts

Pair Programming

Pair Programming is a great way to build something while getting to know someone.

For your upcoming Pair Programming session, you will be the navigator while your CloudZero buddy will be the driver. Please expect your buddy to be a "naive driver", that is he/she is comfortable hands on keyboard and will not do anything without explicit instruction from you.

Domain Driven

The application contained herein will try to solve a real domain problem. This problem is:

1) You have a batch of kinesis data records to process using a lambda function.  This represents a situation where your lambda function is consuming a kinesis data stream.
2) The kinesis data stream contains embedded cloudtrail events of several varieties which will need to be unpacked as the stream records are consumed.
3) Though there are four types of cloudtrail events in the stream, we only are interested in the ones that contain EC2 instances.
4) Extract all of the unique EC2 instance IDs you can find, along with the UTC epoch timestamp (including milliseconds) when the ID was found.
5) Create a CSV file from these pairings of instance ID and timestamp, sorted by timestamp ascending, and upload it to an S3 bucket.

Open

This project is open for your perusal before and after the pair programming session.

Directory Overview

.
├── README.md                   <-- This instructions file
├── event.json                  <-- API Gateway Proxy Integration event payload
├── pairing                     <-- Source code for a lambda function
│   ├── __init__.py
│   ├── app.py                  <-- Lambda function code
│   ├── requirements.txt        <-- Lambda function Requirements
│   ├── sample_data.py          <-- Sample data functions
│   ├── utils.py                <-- Utility functions and classes
├── template.yaml               <-- SAM Template
└── tests                       <-- Unit tests
    └── unit
        ├── __init__.py
        └── test_handler.py

Requirements

• AWS CLI already configured with Administrator permission
• Python 3 installed
• [Docker installed](https://www.docker.com/community-edition**

Setup process

Local development

Initialize

make init

Test

make test

Invoking function locally using a local sample payload

sam local invoke CloudTrailAnalyticsFunction --event event.json

Packaging and deployment

AWS Lambda Python runtime requires a flat folder with all dependencies including the application. SAM will use CodeUri property to know where to look up for both application and dependencies:

...
    CloudTrailAnalyticsFunction:
        Type: AWS::Serverless::Function
        Properties:
            CodeUri: pairing/
            ...

Firstly, we need a S3 bucket where we can upload our Lambda functions packaged as ZIP before we deploy anything - If you don't have a S3 bucket to store code artifacts then this is a good time to create one:

aws s3 mb s3://BUCKET_NAME

Next, run the following command to package our Lambda function to S3:

sam package \
    --output-template-file packaged.yaml \
    --s3-bucket REPLACE_THIS_WITH_YOUR_S3_BUCKET_NAME

Next, the following command will create a Cloudformation Stack and deploy your SAM resources.

sam deploy \
    --template-file packaged.yaml \
    --stack-name sam-app \
    --capabilities CAPABILITY_IAM

See Serverless Application Model (SAM) HOWTO Guide for more details in how to get started.

After deployment is complete you can run the following command to retrieve the API Gateway Endpoint URL:

aws cloudformation describe-stacks \
    --stack-name sam-app \
    --query 'Stacks[].Outputs[?OutputKey==`HelloWorldApi`]' \
    --output table

Fetch, tail, and filter Lambda function logs

To simplify troubleshooting, SAM CLI has a command called sam logs. sam logs lets you fetch logs generated by your Lambda function from the command line. In addition to printing the logs on the terminal, this command has several nifty features to help you quickly find the bug.

NOTE: This command works for all AWS Lambda functions; not just the ones you deploy using SAM.

sam logs -n CloudTrailAnalyticsFunction --stack-name sam-app --tail

You can find more information and examples about filtering Lambda function logs in the SAM CLI Documentation.

Testing

Next, we install test dependencies and we run pytest against our tests folder to run our initial unit tests:

pip install pytest pytest-mock --user
python -m pytest tests/ -v

Cleanup

In order to delete our Serverless Application recently deployed you can use the following AWS CLI Command:

aws cloudformation delete-stack --stack-name sam-app

Bringing to the next level

Here are a few things you can try to get more acquainted with building serverless applications using SAM:

Learn how SAM Build can help you with dependencies

• Uncomment lines on app.py
• Build the project with sam build --use-container
• Invoke with sam local invoke CloudTrailAnalyticsFunction --event event.json
• Update tests

Create an additional API resource

• Create a catch all resource (e.g. /hello/{proxy+}) and return the name requested through this new path
• Update tests

Step-through debugging

• Enable step-through debugging docs for supported runtimes

Next, you can use AWS Serverless Application Repository to deploy ready to use Apps that go beyond hello world samples and learn how authors developed their applications: AWS Serverless Application Repository main page

Appendix

Building the project

AWS Lambda requires a flat folder with the application as well as its dependencies in deployment package. When you make changes to your source code or dependency manifest, run the following command to build your project local testing and deployment:

sam build

If your dependencies contain native modules that need to be compiled specifically for the operating system running on AWS Lambda, use this command to build inside a Lambda-like Docker container instead:

sam build --use-container

By default, this command writes built artifacts to .aws-sam/build folder.

SAM and AWS CLI commands

All commands used throughout this document

# Generate event.json via generate-event command
sam local generate-event apigateway aws-proxy > event.json

# Invoke function locally with event.json as an input
sam local invoke CloudTrailAnalyticsFunction --event event.json

# Run API Gateway locally
sam local start-api

# Create S3 bucket
aws s3 mb s3://BUCKET_NAME

# Package Lambda function defined locally and upload to S3 as an artifact
sam package \
    --output-template-file packaged.yaml \
    --s3-bucket REPLACE_THIS_WITH_YOUR_S3_BUCKET_NAME

# Deploy SAM template as a CloudFormation stack
sam deploy \
    --template-file packaged.yaml \
    --stack-name sam-app \
    --capabilities CAPABILITY_IAM

# Describe Output section of CloudFormation stack previously created
aws cloudformation describe-stacks \
    --stack-name sam-app \
    --query 'Stacks[].Outputs[?OutputKey==`HelloWorldApi`]' \
    --output table

# Tail Lambda function Logs using Logical name defined in SAM Template
sam logs -n CloudTrailAnalyticsFunction --stack-name sam-app --tail