Feature/aws cli in docker by rugeli · Pull Request #280 · mesoscope/cellpack

rugeli · 2024-08-01T21:28:51Z

Problem

What is the problem this work solves, including
closes #279

Solution

What I/we did to solve this problem

Followed the steps in the ticket. We successfully accessed S3 bucket and stored result files and metadata within the Docker container
Updated Readme

Note: We saw two informational warnings while running the packings. These will be addressed in a separate PR.

with @ascibisz

Type of change

Please delete options that are not relevant.

New feature (non-breaking change which adds functionality)

Steps to Verify:

Make sure you have Docker installed locally, and access to cellPACK S3 bucket
Build the container: docker build -t [CONTAINER-NAME] .
Mount AWS credentials into container and run test packing: docker run -v ~/.aws:/root/.aws -e recipe=examples/recipes/v2/one_sphere.json -e config=examples/packing-configs/run.json [CONTAINER-NAME]

You should see output similar to the following in your terminal:

Check metadata is stored in Firebase:

github-actions · 2024-08-01T21:31:48Z

Packing analysis report

Analysis for packing results located at cellpack/tests/outputs/test_spheres/spheresSST

Ingredient name	Encapsulating radius	Average number packed
ext_A	25	236.0

Packing image

Distance analysis

Expected minimum distance: 50.00
Actual minimum distance: 50.01

Ingredient key	Pairwise distance distribution
ext_A

mogres · 2024-08-05T17:36:25Z

README.md

+1. Install [docker](https://docs.docker.com/v17.09/engine/installation/)
+2. Clone the repository locally, if you haven't already: `git clone https://github.com/mesoscope/cellpack.git`
+3. Ensure that you have valid AWS access key and secret to access the `cellpack-results` S3 bucket, usually stored in a `~/.aws/credentials` file. 
+4. To build the container, run: `sudo docker build -t cellpack_image .` Include `--build-arg local=True` if running locally, omit if running on an EC2 instance connected to EFS volume


Does the docker build command need to be run with sudo permissions?

This is a good question! @ascibisz uses sudo docker build -t on her machine whereas I run docker build -t without sudo on mine. I will discuss with Alli how we want to document this difference. Thank you for taking a look!

mogres · 2024-08-05T17:37:23Z

docker-entrypoint.sh

+# fi
+
+cd /cellpack
+# pack -r ${recipe} -c ${config}


Looks like you are taking in the recipe and config parameters as inputs but they are not used here?

Yes, the entry point file is here for us to visualize the docker structure. We will be refining the parameters based on our needs as we discuss with the team

…ture/aws-cli-in-docker

ascibisz · 2024-08-14T16:19:18Z

README.md

+1. Install [docker](https://docs.docker.com/v17.09/engine/installation/)
+2. Clone the repository locally, if you haven't already: `git clone https://github.com/mesoscope/cellpack.git`
+3. Ensure that you have valid AWS access key and secret to access the `cellpack-results` S3 bucket, usually stored in a `~/.aws/credentials` file. 
+4. To build the container, run: `docker build -t cellpack_image .` Include `--build-arg local=True` if running locally, omit if running on an EC2 instance connected to EFS volume


We can remove the second sentence of step 4! This is relevant for octopus, but not needed here :)

* Feature/aws cli in docker (#280) * Add basic Dockerfile and docker-entrypoint file * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme --------- Co-authored-by: ascibisz <alli.scibisz@alleninstitute.org> * Feature/pass recipe to docker (#285) * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme * add parameter for when we're using docker to default to using the staging firebase db * read recipe and config in from docker run params * use dockerignore file to prevent bloat in docker image * add `batch_job_id` to result metadata (#295) * retrieve job id * formatting * add a bucket for batch jobs * formatting * remove print statement and add comments --------- Co-authored-by: Ruge Li <rugeli0605@gmail.com> Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com> * pull changes for firebase configs handling * add docker option to config (#332) * Bug Fix: keep grid_file_path when loading recipe from firebase (#334) * keep grid_file_path when loading recipe from firebase * only add grid_file_path if it's in the firebase recipe * Feature/docker file for ecs (#336) * Add Dockerfile.ecs for our ECS image and rename old Dockerfile to Dockerfile.batch * add intentional error handling for if packing fails * change messaging from /hello endpoint * send response json using camelCase * add healthcheck endpoint and use port 80 instead of 8443 for load balancer requirements * add way for running ECS docker image locally without trying to publish results to cellpack-demo/ --------- Co-authored-by: ascibisz <alli.scibisz@alleninstitute.org> Co-authored-by: Alli <111383930+ascibisz@users.noreply.github.com>

* Add basic Dockerfile and docker-entrypoint file * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme --------- Co-authored-by: ascibisz <alli.scibisz@alleninstitute.org>

* Feature/aws cli in docker (#280) * Add basic Dockerfile and docker-entrypoint file * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme --------- Co-authored-by: ascibisz <alli.scibisz@alleninstitute.org> * Feature/pass recipe to docker (#285) * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme * add parameter for when we're using docker to default to using the staging firebase db * read recipe and config in from docker run params * use dockerignore file to prevent bloat in docker image * add `batch_job_id` to result metadata (#295) * retrieve job id * formatting * add a bucket for batch jobs * formatting * remove print statement and add comments --------- Co-authored-by: Ruge Li <rugeli0605@gmail.com> Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com> * pull changes for firebase configs handling * Feature/docker file for ecs (#336) * Add Dockerfile.ecs for our ECS image and rename old Dockerfile to Dockerfile.batch * add intentional error handling for if packing fails * change messaging from /hello endpoint * add healthcheck endpoint and use port 80 instead of 8443 for load balancer requirements * add way for running ECS docker image locally without trying to publish results to cellpack-demo/ * update server to respond from packing requests immediately, rather than waiting for the packing job to complete * revert accidential change to is_remote_path * remove is_remote flag * lint fix * temporarily use cellpack-results bucket * revert demo bucket name change * change to point cellpack install to main branch --------- Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com> Co-authored-by: Ruge Li <rugeli0605@gmail.com>

* Add basic Dockerfile and docker-entrypoint file * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme --------- Co-authored-by: ascibisz <alli.scibisz@alleninstitute.org>

* tFeature/aws cli in docker (#280) * Add basic Dockerfile and docker-entrypoint file * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme --------- Co-authored-by: ascibisz <alli.scibisz@alleninstitute.org> * Feature/pass recipe to docker (#285) * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme * add parameter for when we're using docker to default to using the staging firebase db * read recipe and config in from docker run params * use dockerignore file to prevent bloat in docker image * add `batch_job_id` to result metadata (#295) * retrieve job id * formatting * add a bucket for batch jobs * formatting * remove print statement and add comments --------- Co-authored-by: Ruge Li <rugeli0605@gmail.com> Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com> * pull changes for firebase configs handling * Feature/docker file for ecs (#336) * Add Dockerfile.ecs for our ECS image and rename old Dockerfile to Dockerfile.batch * add intentional error handling for if packing fails * change messaging from /hello endpoint * add healthcheck endpoint and use port 80 instead of 8443 for load balancer requirements * add way for running ECS docker image locally without trying to publish results to cellpack-demo/ * revert accidential change to is_remote_path * remove is_remote flag * temporarily use cellpack-results bucket * revert demo bucket name change * make changes to cellpack code to accept a remote recipe that closely resembles a local recipe * lint fix * cleanup code * use local variable to improve readability * lint fix --------- Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com> Co-authored-by: Ruge Li <rugeli0605@gmail.com>

ascibisz and others added 4 commits July 11, 2024 15:46

Add basic Dockerfile and docker-entrypoint file

4ea5961

draft aws cli inside of docker container

17f2aa5

access aws s3 with saved cred

055471a

update readme

1461634

rugeli requested review from ascibisz and meganrm August 1, 2024 21:32

mogres approved these changes Aug 6, 2024

View reviewed changes

rugeli added 2 commits August 13, 2024 16:27

Merge branch 'main' of https://github.com/mesoscope/cellpack into fea…

4a2cf51

…ture/aws-cli-in-docker

update docker commands

cc7f7f4

ascibisz reviewed Aug 14, 2024

View reviewed changes

rugeli added 2 commits August 14, 2024 12:00

update readme

626cb68

update readme

e4b8fbb

meganrm approved these changes Aug 15, 2024

View reviewed changes

ascibisz approved these changes Aug 22, 2024

View reviewed changes

rugeli merged commit 081412b into staging/cellpack-server Aug 22, 2024

rugeli deleted the feature/aws-cli-in-docker branch August 22, 2024 18:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

Feature/aws cli in docker#280

Feature/aws cli in docker#280
rugeli merged 8 commits intostaging/cellpack-serverfrom
feature/aws-cli-in-docker

rugeli commented Aug 1, 2024

Uh oh!

github-actions bot commented Aug 1, 2024 •

edited

Loading

Uh oh!

mogres Aug 5, 2024

Uh oh!

rugeli Aug 6, 2024

Uh oh!

mogres Aug 5, 2024

Uh oh!

rugeli Aug 6, 2024

Uh oh!

ascibisz Aug 14, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Comments

Conversation

rugeli commented Aug 1, 2024

Problem

Solution

Type of change

Steps to Verify:

Uh oh!

github-actions bot commented Aug 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Packing analysis report

Analysis for packing results located at cellpack/tests/outputs/test_spheres/spheresSST

Packing image

Distance analysis

Uh oh!

mogres Aug 5, 2024

Choose a reason for hiding this comment

Uh oh!

rugeli Aug 6, 2024

Choose a reason for hiding this comment

Uh oh!

mogres Aug 5, 2024

Choose a reason for hiding this comment

Uh oh!

rugeli Aug 6, 2024

Choose a reason for hiding this comment

Uh oh!

ascibisz Aug 14, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions bot commented Aug 1, 2024 •

edited

Loading