Container for FlowDock

[qhelleu]

Hi,
Thanks for this very interesting tool!
However, the program has many dependencies, to help for installation, could you provide a definition file for Docker or Apptainer or even directly an image in DockerHub?

Thanks a lot.

[amorehead]

Thanks for reaching out! @colbyford, would you be interested in taking this on, since this environment should be similar to that of GCDM?

[colbyford]

@amorehead - Sure! I'll get to work on it tomorrow.

[colbyford]

An update here...

I have a basic Docker image working, but the conda environment is taking far too long to solve as there are over 400 dependencies listed in the flowdock_environment.yaml file.

I'm working on a simplified .yaml file now, which should hopefully fix this issue.

[amorehead]

Thanks, @colbyford! I personally don't recall my environment taking this long to build, but perhaps some of the remote dependencies have changed their installation behavior or versioning since I initially crafted the flowdock_environment.yaml file.

[colbyford]

Apologies for the delay.

I have a working Dockerfile in my fork of the repo, which is in PR4: #4

You can pull my image from Docker Hub and run the sampling logic now:

docker pull cford38/flowdock:latest
docker run --gpus all -v ./checkpoints:/software/flowdock/checkpoints --rm --name flowdock -it cford38/flowdock:latest /bin/bash
python flowdock/sample.py ...

Note that this Docker image does not contain the checkpoints. Thus, you'll need to download the desired checkpoint locally and run the Docker image with the local checkpoint directory mounted as a volume.

[amorehead]

I've merged your PR into main. Thanks again for your invaluable help with setting this image up. @qhelleu, does @colbyford 's latest additions address your issue?

[qhelleu]

Hi,
It doesn't work on my side:
After pulling the container with apptainer:
apptainer pull docker://cford38/flowdock:latest

I run a check command which doesn't work:
apptainer exec --nv $DIR/bin/flowdock_latest.sif python sample.py
/opt/conda/bin/python: can't open file '/home/***/FlowDock/sample.py': [Errno 2] No such file or directory

Then I try with the path inside the container, sample.py runs but the packages are not found:
apptainer exec --nv $DIR/bin/flowdock_latest.sif python /software/flowdock/flowdock/sample.py
Traceback (most recent call last):
File "/software/flowdock/flowdock/sample.py", line 3, in
import hydra
ModuleNotFoundError: No module named 'hydra'

Then I try calling the command from inside the conda env, but still not working:
apptainer exec --nv $DIR/bin/flowdock_latest.sif conda run python /software/flowdock/flowdock/sample.py
/tmp/tmpmmtsdoy5: line 1: /usr/local/bin/conda/bin/conda: No such file or directory
usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...
conda: error: argument COMMAND: invalid choice: 'activate' (choose from 'clean', 'compare', 'config', 'create', 'info', 'init', 'install', 'list', 'notices', 'package', 'remove', 'uninstall', 'rename', 'run', 'search', 'update', 'upgrade', 'build', 'content-trust', 'convert', 'debug', 'develop', 'doctor', 'index', 'inspect', 'metapackage', 'render', 'skeleton', 'env')
Traceback (most recent call last):
File "/software/flowdock/flowdock/sample.py", line 3, in
import hydra
ModuleNotFoundError: No module named 'hydra'

[colbyford]

Good morning!

There seems to be some things being ignored in the Apptainer/Singularity converted version of the Docker image.

• The conda environment isn't automatically activating as it does in the Docker image I created.
• The default working directory, /software/flowdock/, is being ignored.

To fix this, you'll need to change the directory and activate the flowdock conda environment before you execute the sample.py script.

You could create a bash script like this:

#!/bin/bash

cd /software/flowdock

source activate flowdock
python flowdock/sample.py ...

Then,

singularity exec flowdock_latest.sif /bin/bash run_flowdock.sh

[amorehead]

Hi, @qhelleu. Does @colbyford's response solve your issue?

[qhelleu]

Hi,
Using Mamba I managed to make it work without a script.
Here is the definition file I'm using (it's based on esmfold definition file):
FlowDock_v0.0.2-cudnn8-devel-ubuntu18.04.txt
Thanks,
Best.

[amorehead]

Thanks for sharing how you got it to work on your end, @qhelleu!