Feature/pass recipe to docker by ascibisz · Pull Request #285 · mesoscope/cellpack

ascibisz · 2024-08-22T17:21:34Z

Problem

Cleaning up docker code so we are actually passing the recipe / config parameters specified in the commandline / as ENV variables into the docker container.
Link to story or ticket

Solution

Updated docker-entrypoint.sh to actually read provided recipe and config file paths and pass them to the pack script
Added .dockerignore file so we aren't copying unneeded files into the docker image
Added an optional boolean docker for when we want to default to pointing to staging for firebase without prompting the user via commandline. To use, we can call the pack script from the commandline with the -d flag, i.e. pack -r my_recipe.json -c my_config.json -d

Type of change

New feature (non-breaking change which adds functionality)

github-actions · 2024-08-22T17:24:34Z

Packing analysis report

Analysis for packing results located at cellpack/tests/outputs/test_spheres/spheresSST

Ingredient name	Encapsulating radius	Average number packed
ext_A	25	236.0

Packing image

Distance analysis

Expected minimum distance: 50.00
Actual minimum distance: 50.01

Ingredient key	Pairwise distance distribution
ext_A

…ging firebase db

rugeli

LGTM! We can update the documentation for triggering batch jobs once the gateway api is in place. If we want to rename variables like use_docker, we can handle that in a cleanup pr before merging into main from staging.

mogres

I was able to build the image but could not run it on the on-site dgx machines due to docker being unable to access my home directory. However, the implementation looks good to me!

mogres · 2024-09-20T18:54:16Z

cellpack/autopack/__init__.py



-def load_file(filename, destination="", cache="geometries", force=None):
+def load_file(filename, destination="", cache="geometries", force=None, use_docker=False):


Could we make this argument name more generic? Seeing as this flag decides the database to use, maybe it could be renamed to skip_database_check or something equivalent?

* retrieve job id * formatting * add a bucket for batch jobs * formatting * remove print statement and add comments

* Feature/aws cli in docker (#280) * Add basic Dockerfile and docker-entrypoint file * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme --------- Co-authored-by: ascibisz <alli.scibisz@alleninstitute.org> * Feature/pass recipe to docker (#285) * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme * add parameter for when we're using docker to default to using the staging firebase db * read recipe and config in from docker run params * use dockerignore file to prevent bloat in docker image * add `batch_job_id` to result metadata (#295) * retrieve job id * formatting * add a bucket for batch jobs * formatting * remove print statement and add comments --------- Co-authored-by: Ruge Li <rugeli0605@gmail.com> Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com> * pull changes for firebase configs handling * add docker option to config (#332) * Bug Fix: keep grid_file_path when loading recipe from firebase (#334) * keep grid_file_path when loading recipe from firebase * only add grid_file_path if it's in the firebase recipe * Feature/docker file for ecs (#336) * Add Dockerfile.ecs for our ECS image and rename old Dockerfile to Dockerfile.batch * add intentional error handling for if packing fails * change messaging from /hello endpoint * send response json using camelCase * add healthcheck endpoint and use port 80 instead of 8443 for load balancer requirements * add way for running ECS docker image locally without trying to publish results to cellpack-demo/ --------- Co-authored-by: ascibisz <alli.scibisz@alleninstitute.org> Co-authored-by: Alli <111383930+ascibisz@users.noreply.github.com>

* draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme * add parameter for when we're using docker to default to using the staging firebase db * read recipe and config in from docker run params * use dockerignore file to prevent bloat in docker image * add `batch_job_id` to result metadata (#295) * retrieve job id * formatting * add a bucket for batch jobs * formatting * remove print statement and add comments --------- Co-authored-by: Ruge Li <rugeli0605@gmail.com> Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com>

* Feature/aws cli in docker (#280) * Add basic Dockerfile and docker-entrypoint file * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme --------- Co-authored-by: ascibisz <alli.scibisz@alleninstitute.org> * Feature/pass recipe to docker (#285) * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme * add parameter for when we're using docker to default to using the staging firebase db * read recipe and config in from docker run params * use dockerignore file to prevent bloat in docker image * add `batch_job_id` to result metadata (#295) * retrieve job id * formatting * add a bucket for batch jobs * formatting * remove print statement and add comments --------- Co-authored-by: Ruge Li <rugeli0605@gmail.com> Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com> * pull changes for firebase configs handling * Feature/docker file for ecs (#336) * Add Dockerfile.ecs for our ECS image and rename old Dockerfile to Dockerfile.batch * add intentional error handling for if packing fails * change messaging from /hello endpoint * add healthcheck endpoint and use port 80 instead of 8443 for load balancer requirements * add way for running ECS docker image locally without trying to publish results to cellpack-demo/ * update server to respond from packing requests immediately, rather than waiting for the packing job to complete * revert accidential change to is_remote_path * remove is_remote flag * lint fix * temporarily use cellpack-results bucket * revert demo bucket name change * change to point cellpack install to main branch --------- Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com> Co-authored-by: Ruge Li <rugeli0605@gmail.com>

* draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme * add parameter for when we're using docker to default to using the staging firebase db * read recipe and config in from docker run params * use dockerignore file to prevent bloat in docker image * add `batch_job_id` to result metadata (#295) * retrieve job id * formatting * add a bucket for batch jobs * formatting * remove print statement and add comments --------- Co-authored-by: Ruge Li <rugeli0605@gmail.com> Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com>

* tFeature/aws cli in docker (#280) * Add basic Dockerfile and docker-entrypoint file * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme --------- Co-authored-by: ascibisz <alli.scibisz@alleninstitute.org> * Feature/pass recipe to docker (#285) * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme * add parameter for when we're using docker to default to using the staging firebase db * read recipe and config in from docker run params * use dockerignore file to prevent bloat in docker image * add `batch_job_id` to result metadata (#295) * retrieve job id * formatting * add a bucket for batch jobs * formatting * remove print statement and add comments --------- Co-authored-by: Ruge Li <rugeli0605@gmail.com> Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com> * pull changes for firebase configs handling * Feature/docker file for ecs (#336) * Add Dockerfile.ecs for our ECS image and rename old Dockerfile to Dockerfile.batch * add intentional error handling for if packing fails * change messaging from /hello endpoint * add healthcheck endpoint and use port 80 instead of 8443 for load balancer requirements * add way for running ECS docker image locally without trying to publish results to cellpack-demo/ * revert accidential change to is_remote_path * remove is_remote flag * temporarily use cellpack-results bucket * revert demo bucket name change * make changes to cellpack code to accept a remote recipe that closely resembles a local recipe * lint fix * cleanup code * use local variable to improve readability * lint fix --------- Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com> Co-authored-by: Ruge Li <rugeli0605@gmail.com>

Base automatically changed from feature/aws-cli-in-docker to staging/cellpack-server August 22, 2024 18:24

rugeli and others added 9 commits August 26, 2024 13:14

draft aws cli inside of docker container

6d5f130

access aws s3 with saved cred

807ba73

update readme

1284bb7

update docker commands

ef2f845

update readme

146ce0f

update readme

5406abd

add parameter for when we're using docker to default to using the sta…

235aed2

…ging firebase db

read recipe and config in from docker run params

f88286b

use dockerignore file to prevent bloat in docker image

3480a55

ascibisz force-pushed the feature/pass-recipe-to-docker branch from 9d74276 to 3480a55 Compare August 26, 2024 20:15

ascibisz marked this pull request as ready for review September 18, 2024 20:21

ascibisz requested review from meganrm, mogres and rugeli September 18, 2024 20:21

rugeli approved these changes Sep 20, 2024

View reviewed changes

mogres approved these changes Sep 20, 2024

View reviewed changes

meganrm removed their request for review September 23, 2024 22:42

add batch_job_id to result metadata (#295)

cdef941

* retrieve job id * formatting * add a bucket for batch jobs * formatting * remove print statement and add comments

rugeli merged commit b5e4fa0 into staging/cellpack-server Nov 15, 2024

rugeli deleted the feature/pass-recipe-to-docker branch November 15, 2024 19:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

Feature/pass recipe to docker#285

Feature/pass recipe to docker#285
rugeli merged 10 commits intostaging/cellpack-serverfrom
feature/pass-recipe-to-docker

ascibisz commented Aug 22, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Aug 22, 2024 •

edited

Loading

Uh oh!

rugeli left a comment

Uh oh!

mogres left a comment

Uh oh!

mogres Sep 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants



		def load_file(filename, destination="", cache="geometries", force=None):
		def load_file(filename, destination="", cache="geometries", force=None, use_docker=False):

Comments

Conversation

ascibisz commented Aug 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Type of change

Uh oh!

github-actions bot commented Aug 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Packing analysis report

Analysis for packing results located at cellpack/tests/outputs/test_spheres/spheresSST

Packing image

Distance analysis

Uh oh!

rugeli left a comment

Choose a reason for hiding this comment

Uh oh!

mogres left a comment

Choose a reason for hiding this comment

Uh oh!

mogres Sep 20, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ascibisz commented Aug 22, 2024 •

edited

Loading

github-actions bot commented Aug 22, 2024 •

edited

Loading