add `batch_job_id` to result metadata by rugeli · Pull Request #295 · mesoscope/cellpack

rugeli · 2024-10-28T20:28:43Z

Problem

What is the problem this work solves, including
closes #296

Solution

What I/we did to solve this problem

batch job ids are stored as environment tokens, we can retrieve them via os
add a batch_job_id field to result metadata, which will be uploaded to firebase upon successful packing. If the packing is local, the value will be null

with @ascibisz

Type of change

Please delete options that are not relevant.

New feature (non-breaking change which adds functionality)

Steps to Verify:

use API gateway to trigger a packing batch job (currently, only Alli and Ruge have access, but I'll attach screenshots to demonstrate the steps )
wait 30-60sec for the job to run
check the results collection in staging firebase database - recipe metadata should display the correct batch job id
optional: pack a recipe locally - the batch_job_id field should also be present but with a null value

Screenshots (optional):

Show-n-tell images/animations here
Step 1 (API Gateway): use API gateway to trigger a packing batch job

Step 2 (AWS Batch): wait for the batch job to run

Step 3 (Firebase): batch job id will be added to the metadata and uploaded to firebase

github-actions · 2024-10-28T20:32:00Z

Packing analysis report

Analysis for packing results located at cellpack/tests/outputs/test_spheres/spheresSST

Ingredient name	Encapsulating radius	Average number packed
ext_A	25	236.0

Packing image

Distance analysis

Expected minimum distance: 50.00
Actual minimum distance: 50.01

Ingredient key	Pairwise distance distribution
ext_A

cellpack/autopack/upy/simularium/simularium_helper.py

ascibisz

LGTM!

* draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme * add parameter for when we're using docker to default to using the staging firebase db * read recipe and config in from docker run params * use dockerignore file to prevent bloat in docker image * add `batch_job_id` to result metadata (#295) * retrieve job id * formatting * add a bucket for batch jobs * formatting * remove print statement and add comments --------- Co-authored-by: Ruge Li <rugeli0605@gmail.com> Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com>

* Feature/aws cli in docker (#280) * Add basic Dockerfile and docker-entrypoint file * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme --------- Co-authored-by: ascibisz <alli.scibisz@alleninstitute.org> * Feature/pass recipe to docker (#285) * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme * add parameter for when we're using docker to default to using the staging firebase db * read recipe and config in from docker run params * use dockerignore file to prevent bloat in docker image * add `batch_job_id` to result metadata (#295) * retrieve job id * formatting * add a bucket for batch jobs * formatting * remove print statement and add comments --------- Co-authored-by: Ruge Li <rugeli0605@gmail.com> Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com> * pull changes for firebase configs handling * add docker option to config (#332) * Bug Fix: keep grid_file_path when loading recipe from firebase (#334) * keep grid_file_path when loading recipe from firebase * only add grid_file_path if it's in the firebase recipe * Feature/docker file for ecs (#336) * Add Dockerfile.ecs for our ECS image and rename old Dockerfile to Dockerfile.batch * add intentional error handling for if packing fails * change messaging from /hello endpoint * send response json using camelCase * add healthcheck endpoint and use port 80 instead of 8443 for load balancer requirements * add way for running ECS docker image locally without trying to publish results to cellpack-demo/ --------- Co-authored-by: ascibisz <alli.scibisz@alleninstitute.org> Co-authored-by: Alli <111383930+ascibisz@users.noreply.github.com>

* draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme * add parameter for when we're using docker to default to using the staging firebase db * read recipe and config in from docker run params * use dockerignore file to prevent bloat in docker image * add `batch_job_id` to result metadata (#295) * retrieve job id * formatting * add a bucket for batch jobs * formatting * remove print statement and add comments --------- Co-authored-by: Ruge Li <rugeli0605@gmail.com> Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com>

* Feature/aws cli in docker (#280) * Add basic Dockerfile and docker-entrypoint file * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme --------- Co-authored-by: ascibisz <alli.scibisz@alleninstitute.org> * Feature/pass recipe to docker (#285) * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme * add parameter for when we're using docker to default to using the staging firebase db * read recipe and config in from docker run params * use dockerignore file to prevent bloat in docker image * add `batch_job_id` to result metadata (#295) * retrieve job id * formatting * add a bucket for batch jobs * formatting * remove print statement and add comments --------- Co-authored-by: Ruge Li <rugeli0605@gmail.com> Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com> * pull changes for firebase configs handling * Feature/docker file for ecs (#336) * Add Dockerfile.ecs for our ECS image and rename old Dockerfile to Dockerfile.batch * add intentional error handling for if packing fails * change messaging from /hello endpoint * add healthcheck endpoint and use port 80 instead of 8443 for load balancer requirements * add way for running ECS docker image locally without trying to publish results to cellpack-demo/ * update server to respond from packing requests immediately, rather than waiting for the packing job to complete * revert accidential change to is_remote_path * remove is_remote flag * lint fix * temporarily use cellpack-results bucket * revert demo bucket name change * change to point cellpack install to main branch --------- Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com> Co-authored-by: Ruge Li <rugeli0605@gmail.com>

* tFeature/aws cli in docker (#280) * Add basic Dockerfile and docker-entrypoint file * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme --------- Co-authored-by: ascibisz <alli.scibisz@alleninstitute.org> * Feature/pass recipe to docker (#285) * draft aws cli inside of docker container * access aws s3 with saved cred * update readme * update docker commands * update readme * update readme * add parameter for when we're using docker to default to using the staging firebase db * read recipe and config in from docker run params * use dockerignore file to prevent bloat in docker image * add `batch_job_id` to result metadata (#295) * retrieve job id * formatting * add a bucket for batch jobs * formatting * remove print statement and add comments --------- Co-authored-by: Ruge Li <rugeli0605@gmail.com> Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com> * pull changes for firebase configs handling * Feature/docker file for ecs (#336) * Add Dockerfile.ecs for our ECS image and rename old Dockerfile to Dockerfile.batch * add intentional error handling for if packing fails * change messaging from /hello endpoint * add healthcheck endpoint and use port 80 instead of 8443 for load balancer requirements * add way for running ECS docker image locally without trying to publish results to cellpack-demo/ * revert accidential change to is_remote_path * remove is_remote flag * temporarily use cellpack-results bucket * revert demo bucket name change * make changes to cellpack code to accept a remote recipe that closely resembles a local recipe * lint fix * cleanup code * use local variable to improve readability * lint fix --------- Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com> Co-authored-by: Ruge Li <rugeli0605@gmail.com>

rugeli added 4 commits October 22, 2024 14:45

retrieve job id

4071e47

formatting

93324a6

add a bucket for batch jobs

685d4ca

formatting

19d556c

rugeli changed the title ~~Feature/job~~ add batch_job_id to result metadata Oct 28, 2024

rugeli requested review from ascibisz and meganrm October 28, 2024 22:35

rugeli commented Oct 28, 2024

View reviewed changes

cellpack/autopack/upy/simularium/simularium_helper.py Show resolved Hide resolved

ascibisz reviewed Nov 1, 2024

View reviewed changes

cellpack/autopack/upy/simularium/simularium_helper.py Outdated Show resolved Hide resolved

remove print statement and add comments

e82ca6f

meganrm approved these changes Nov 5, 2024

View reviewed changes

ascibisz approved these changes Nov 12, 2024

View reviewed changes

rugeli merged commit cdef941 into feature/pass-recipe-to-docker Nov 12, 2024

rugeli deleted the feature/job-id branch November 12, 2024 22:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

add `batch_job_id` to result metadata#295

add `batch_job_id` to result metadata#295
rugeli merged 5 commits intofeature/pass-recipe-to-dockerfrom
feature/job-id

rugeli commented Oct 28, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Oct 28, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

ascibisz left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Conversation

rugeli commented Oct 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Solution

Type of change

Steps to Verify:

Screenshots (optional):

Uh oh!

github-actions bot commented Oct 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Packing analysis report

Analysis for packing results located at cellpack/tests/outputs/test_spheres/spheresSST

Packing image

Distance analysis

Uh oh!

Uh oh!

Uh oh!

ascibisz left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rugeli commented Oct 28, 2024 •

edited

Loading

github-actions bot commented Oct 28, 2024 •

edited

Loading