1. Login to your AWS Account.
2. Go to AWS IAM (Identity and Access Management) Service.
3. Security Credentials > Create access key > Use Case - Command Line Interface CLI
4. Select the Next Option and Type a Description for your Key. For Example: gtex_pipeline
5. You now have an Access Key ID and a Secret Access Key that we will use to configure AWS. Download .csv file.
6. Open ubuntu terminal and run the following commands:
sudo apt-get update # update packages
sudo apt install awscli # install aws command line interface
aws configure # configure settings that aws-cli will use to interact with aws
Type in your Access Key ID, Secret Access Key, Default Region Name (us-east-1), Default Output Format (json, text or table)
7. You have successfully configured your AWS-CLI to AWS. To verify your configuration run:
aws configure list
8. To test the configuration you can run a simple commnd to list your S3 Buckets:
aws s3 ls
1. Open Ubuntu and run the following commands:
docker image build -t gtex_pipeline/primary:1.0 . # building an image. {-t <\ tag-name>\ . }
The last period (.) means the Dockerfile is present in the current directory.
Message: -- Successfully built db7d00a0bbea. This baseID can be used to access the Docker image instead of using its name tag.
docker image ls # verify your image
2. Try running your docker image in your local system
docker run -p 8000:8000 gtex_pipeline/primary:1.0
3. Push this image to a container resgistry. Here we use Amazon Elastic Container Service (ECS)
a. First begin with creating a repository in ECR. (Create a place to push our image).
aws ecr create-repository --repository-name gtex_pipeline --region us-east-1
This repository creation can be confirmed by going to AWS >> Services >> ECR >> Repositories.
Check your repo name. This will be empty because we don't have any images in it yet.
Also note that AWS will assign a unique URI to your repository. Save that URI.
b. Authenticate Docker credentials with AWS and get an authentication token
aws ecr get-login-password --region us-east-1
We should have two things saved before we move forward. The token we created above and the URI
aws ecr --region us-east-1 | docker login -u AWS -p <encrypted_token> <ecr-repo_uri>
-u: default user provided by AWS, -p: password we retried in the last step. repo-uri: URI of our repository.
c. Tag our local docker image with our ECR Repository.
docker tag gtex_pipeline/primary:1.0 <ecr-repo_uri>
gtex_pipeline/primary:1.0 is the tag we provided while building our docker image.
d. Finally, we push the docker image to ECR.
docker push <ecr-repo-uri> # Docker can be viewed by going into the ECR repo
-
Pipeline Overview
-
Optimatization Results
