Skip to content

Comments

Staging branch: containerization#333

Merged
ascibisz merged 13 commits intomainfrom
staging/cellpack-server
Jun 17, 2025
Merged

Staging branch: containerization#333
ascibisz merged 13 commits intomainfrom
staging/cellpack-server

Conversation

@rugeli
Copy link
Collaborator

@rugeli rugeli commented May 7, 2025 •
edited
Loading

Problem

What is the problem this work solves, including
Dockerization of cellpack codebase

Solution

What I/we did to solve this problem

  • added docker-related files: Dockerfile, docker-entrypoint.sh, dockerignore
  • added -d flag to set staging database as default for client-end jobs
  • notified S3 bucket to store batch job IDs when applicable
  • updated README with docker usage instructions

next TODO: #323 (currently blocking because our AWS role is temporary with short token session, will get this fix when IT replies). We are merging this branch slightly ahead of time to make the ECS infrastructure setup easier.

shout out to @ascibisz for awesome system design insights and collaborations!

Type of change

Please delete options that are not relevant.

  • New feature (non-breaking change which adds functionality)

Steps to Verify:

  1. The primary purpose is to provide Docker images for Batch and ECS - no direct testing needed in this repo

rugeli and others added 8 commits August 22, 2024 11:24
@rugeli @ascibisz
Feature/aws cli in docker (#280)
081412b 
* Add basic Dockerfile and docker-entrypoint file

* draft aws cli inside of docker container

* access aws s3 with saved cred

* update readme

* update docker commands

* update readme

* update readme

---------

Co-authored-by: ascibisz <alli.scibisz@alleninstitute.org>
@rugeli
Merge branch 'main' of https://github.com/mesoscope/cellpack into sta…
5218b5d 
…ging/cellpack-server
@ascibisz @rugeli
Feature/pass recipe to docker (#285)
b5e4fa0 
* draft aws cli inside of docker container

* access aws s3 with saved cred

* update readme

* update docker commands

* update readme

* update readme

* add parameter for when we're using docker to default to using the staging firebase db

* read recipe and config in from docker run params

* use dockerignore file to prevent bloat in docker image

* add `batch_job_id` to result metadata (#295)

* retrieve job id

* formatting

* add a bucket for batch jobs

* formatting

* remove print statement and add comments

---------

Co-authored-by: Ruge Li <rugeli0605@gmail.com>
Co-authored-by: Ruge Li <91452427+rugeli@users.noreply.github.com>
@rugeli
Merge branch 'staging/cellpack-server' of https://github.com/mesoscop…
7866581 
…e/cellpack into staging/cellpack-server
@rugeli
Merge branch 'main' of https://github.com/mesoscope/cellpack into sta…
035b7ba 
…ging/cellpack-server
@rugeli
Merge branch 'main' of https://github.com/mesoscope/cellpack into sta…
6e8b834 
…ging/cellpack-server
@rugeli
pull changes for firebase configs handling
949d681
@rugeli
add docker option to config (#332)
3ee623f
@rugeli rugeli requested review from ascibisz and meganrm May 7, 2025 23:39
@github-actions
Copy link
Contributor

github-actions bot commented May 7, 2025 •
edited
Loading

Packing analysis report

Analysis for packing results located at cellpack/tests/outputs/test_spheres/spheresSST

Ingredient name Encapsulating radius Average number packed
ext_A 25 236.0

Packing image

Packing image

Distance analysis

Expected minimum distance: 50.00
Actual minimum distance: 50.01

Ingredient key Pairwise distance distribution
ext_A Distance distribution ext_A

@rugeli rugeli changed the title Merging server related code from staging branch Staging branch: containerization May 8, 2025
ascibisz added 2 commits May 8, 2025 16:33
@ascibisz
Bug Fix: keep grid_file_path when loading recipe from firebase (#334)
83b240b 
* keep grid_file_path when loading recipe from firebase

* only add grid_file_path if it's in the firebase recipe
@ascibisz
Feature/docker file for ecs (#336)
de4b4fa 
* Add Dockerfile.ecs for our ECS image and rename old Dockerfile to Dockerfile.batch

* add intentional error handling for if packing fails

* change messaging from /hello endpoint
@rugeli rugeli requested a review from mogres May 15, 2025 23:32
mogres
mogres reviewed May 20, 2025
View reviewed changes
Copy link
Collaborator

@mogres mogres left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me!!

Tested by running a test packing on batch via docker. The README instructions were easy to follow! The image was fairly large and took a long time to setup using pip. At some point we should revisit pruning the requirements, pinning versions, adding a pyproject.toml, upgrading the python version, and using a modern package manager like uv or pdm for cellPACK. Maybe this can be something that can be tackled early/mid June?

@mogres
Copy link
Collaborator

mogres commented May 20, 2025 •
edited by rugeli
Loading

Having an issue with uploading results through the dockerfile right now. Uploads work on my local machine but not when run through the dockerfile.

boto3.exceptions.S3UploadFailedError: Failed to upload out/one_sphere/jitter/results_one_sphere_default-run_1.0.0_seed_0.simularium to cellpack-results/simularium/results_one_sphere_default-run_1.0.0_seed_0.simularium: An error occurred (AccessDenied) when calling the PutObject operation: Access Deniedbash

@rugeli
Copy link
Collaborator Author

rugeli commented May 20, 2025

Looks good to me!!

Tested by running a test packing on batch via docker. The README instructions were easy to follow! The image was fairly large and took a long time to setup using pip. At some point we should revisit pruning the requirements, pinning versions, adding a pyproject.toml, upgrading the python version, and using a modern package manager like uv or pdm for cellPACK. Maybe this can be something that can be tackled early/mid June?

Thanks for testing! We're focused on manuscript writing, ECS setup, and UX designs, but everything you mentioned above is definitely should be on our development checklist.

@rugeli
Copy link
Collaborator Author

rugeli commented May 21, 2025

Having an issue with uploading results through the dockerfile right now. Uploads work on my local machine but not when run through the dockerfile.

boto3.exceptions.S3UploadFailedError: Failed to upload out/one_sphere/jitter/results_one_sphere_default-run_1.0.0_seed_0.simularium to cellpack-results/simularium/results_one_sphere_default-run_1.0.0_seed_0.simularium: An error occurred (AccessDenied) when calling the PutObject operation: Access Deniedbash

Yeah that's expected for now, we're using a different aws role for docker-related work, and are still working with IT to getting a more permanent aws setup. We'll keep the team posted once we have a solution!

@mogres
Copy link
Collaborator

mogres commented May 21, 2025

Yeah that's expected for now, we're using a different aws role for docker-related work, and are still working with IT to getting a more permanent aws setup. We'll keep the team posted once we have a solution!

Thanks for the clarification! Does this also apply when running Docker locally? The instructions map my local ~/.aws to the docker file which has valid credentials.

@ascibisz ascibisz merged commit 6b9570f into main Jun 17, 2025
7 checks passed
@ascibisz ascibisz deleted the staging/cellpack-server branch June 17, 2025 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mogres mogres mogres left review comments

@meganrm meganrm meganrm approved these changes

@ascibisz ascibisz ascibisz approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@rugeli @mogres @meganrm @ascibisz