Simplifying the installation process?

[JaGeo]

I think a huge problem with using tools like jobflow-remote is still the installation process.

Do you have any ideas how we could simplify this?

Is there a way to provide a conda environment/a docker including database installation (maybe even with a default setup of the database)?

We just got another round of questions on the atomate2 repo.

[Andrew-S-Rosen]

A docker container is an interesting idea. I think the biggest challenge is the database. If someone were able to code up a file-based Maggma store (even if it is not performant!) that supports parallel processes as noted in materialsproject/maggma#832, it would be much easier to describe the installation process. But as it stands, by essentially relying on MongoDB, the barrier for entry will remain high unless it can be solved with a container.

I will note that this is something Prefect does very well. They ship everything you need in a Conda environment or as a container, and it "just works." The default database configuration is not very performant, but for testing purposes, it works just fine.

[davidwaroquiers]

I think it would indeed be a good idea to have something like that. It would definitely lower the barrier to entry when someone wants to try it out first. There is one thing I would do though is to explicitly add a warning all the time when you are using such less-performant store (or implementation - currently jobflow-remote requires a real mongodb database because it uses aggregations, which are not supported by current file-based Stores - I don't think it would ever be btw). Otherwise some users may think "oh great, I don't need a database in production". I think anyone serious about doing high-throughput production should, after having tested with the less-performant store, should take the time to set his production environment up with a real database.

Warning could be:
"You are using a mock database with limited performance and reliability. This database is provided only for testing and training purposes. For production, you must set up a real MongoDB database as explained ."

I'm not necessarily in favor of a container. It can be quite some maintenance headache, the "it just works" promise is not always (duh) satisfied.

I think it would be possible to have a fallback implementation that does not use aggregations (@gpetretto ?) but with 1/ limited performand (of course) and 2/ (more important) limited reliability because of possible concurrent accesses. (hence the warning that should be there all the time)

What do you think ?

[JaGeo]

I think a typical use-case that we do not cover yet:
I want to run one elastic or one phonon workflow for a large structure but I don't want to setup/configure a database myself for this.

[gpetretto]

Hi,
thanks for raising this issue.
I actually think that there are several points where a user may get lost in setting things up. Unfortunately, I don't think it will be easy to provide something ready to use for all of them, but at least we can try to make it easier.

For the database, indeed we could prepare a further simplified version of the docker container in https://github.com/Matgenix/jobflow_tutorial, removing the local worker and just leaving the runner machine plus MongoDB. However, this will work only if they can run jobflow-remote on a local workstation. If they want to have the runner on the front-end of the cluster they will either need to be able to reach the DB from the cluster (which likely does not include a personal workstation where docker can be executed) or running the DB directly on the cluster frontend (which I would not really recommend for a beginner. There is definitely a possibility that cluster administrators will complain or to expose the data to other users, if the DB is left without any authentication).
We could prepare a predefined configuration for the free tier of MognoDB Atlas, but this could only work for a short amount of time, as performances are not very good and the disk space will be over quite soon.
As already mentioned, an alternative solution would be to devise a way to use file-based DB, but this is not so trivial. I keep thinking about it and the problem is mainly to identify the right tool to use, that will not be abandoned in a short time. I am still making some tests on this.

However, once the DB is set, one still needs to perform some minimal setup of the project configuration to point to the worker. A thing that is strictly dependent on the user HPC system. @Andrew-S-Rosen, I suppose that this is true for Prefect as well. Or do you really have the feeling that you can easily run on HPC systems out of the box? In that case, how is that achieved?
For jobflow-remote we can maybe provide some tool to automatically fill in a project configuration file? But will still require some work from the user.

Finally, a user needs to understand how to actually use the tool: submitting a Flow properly with the correct resources, monitor the status of the Jobs, extracting results and maybe even check the errors when something goes wrong. My experience with newcomers to workflow systems is that learning how to perform these actions is still something that generate a lot of doubts, even after providing individual training.
I don't think this is strictly related to jobflow-remote, as I had the same kind of experience with new Fireworks users and even on myself when I have used Aiida. Despite all the experience, the differences in setup/approach/commands required me to consult the documentation quite often.

My main point here is that we should definitely try to make things smoother for the generic user, even if I doubt we will ever manage to make something as simple as the run_locally within jobflow-remote or fireworks. But it should be somewhat clear that the user should be available to put in some efforts in learning the basics of the tools. I have the feeling that scientific workflows are often sold as an effortless way of running the simulations and thus some users may end up having the wrong expectations of the amount of work they will still need to put in learning how to properly use them.

[Andrew-S-Rosen]

@Andrew-S-Rosen, I suppose that this is true for Prefect as well. Or do you really have the feeling that you can easily run on HPC systems out of the box? In that case, how is that achieved?

You are correct. The interfacing with the job scheduler remains a learning curve for every single one of the 300+ workflow systems published to date, including Prefect (it uses dask-jobqueue to interact with the scheduler, which is terrible).

[JaGeo]

Hi, thanks for raising this issue. I actually think that there are several points where a user may get lost in setting things up. Unfortunately, I don't think it will be easy to provide something ready to use for all of them, but at least we can try to make it easier.

For the database, indeed we could prepare a further simplified version of the docker container in https://github.com/Matgenix/jobflow_tutorial, removing the local worker and just leaving the runner machine plus MongoDB. However, this will work only if they can run jobflow-remote on a local workstation. If they want to have the runner on the front-end of the cluster they will either need to be able to reach the DB from the cluster (which likely does not include a personal workstation where docker can be executed) or running the DB directly on the cluster frontend (which I would not really recommend for a beginner. There is definitely a possibility that cluster administrators will complain or to expose the data to other users, if the DB is left without any authentication). We could prepare a predefined configuration for the free tier of MognoDB Atlas, but this could only work for a short amount of time, as performances are not very good and the disk space will be over quite soon. As already mentioned, an alternative solution would be to devise a way to use file-based DB, but this is not so trivial. I keep thinking about it and the problem is mainly to identify the right tool to use, that will not be abandoned in a short time. I am still making some tests on this.

However, once the DB is set, one still needs to perform some minimal setup of the project configuration to point to the worker. A thing that is strictly dependent on the user HPC system. @Andrew-S-Rosen, I suppose that this is true for Prefect as well. Or do you really have the feeling that you can easily run on HPC systems out of the box? In that case, how is that achieved? For jobflow-remote we can maybe provide some tool to automatically fill in a project configuration file? But will still require some work from the user.

Finally, a user needs to understand how to actually use the tool: submitting a Flow properly with the correct resources, monitor the status of the Jobs, extracting results and maybe even check the errors when something goes wrong. My experience with newcomers to workflow systems is that learning how to perform these actions is still something that generate a lot of doubts, even after providing individual training. I don't think this is strictly related to jobflow-remote, as I had the same kind of experience with new Fireworks users and even on myself when I have used Aiida. Despite all the experience, the differences in setup/approach/commands required me to consult the documentation quite often.

My main point here is that we should definitely try to make things smoother for the generic user, even if I doubt we will ever manage to make something as simple as the run_locally within jobflow-remote or fireworks. But it should be somewhat clear that the user should be available to put in some efforts in learning the basics of the tools. I have the feeling that scientific workflows are often sold as an effortless way of running the simulations and thus some users may end up having the wrong expectations of the amount of work they will still need to put in learning how to properly use them.

I think I would cover these two scenarios with a simplified Installation (ideally with a installation script)

• local workstation with MongoDB (preferred locally) that can access a remote cluster

• Installation on the cluster frontend

A local linux workstation is likely the easiest thing to configure for people and for non-high-throughput works it should be fine.

I agree with the other points but tutorials will help there. After all, we are talking about people that figured out how to run/compile DFT codes on HPCs

[gpetretto]

Hi all,

I have an update concerning the possibility of making things easier for beginners and I would like to hear what you think about it.

I have quickly "vibe coded" an alternative version of JobController that relies on SQL and can use SQLite as a backend instead of MongoDB. This, together with a file based maggma store, like the MontyStore, allows to run a version of jobflow-remote entirely based on files, with no DB server required.
Most of the tests pass, but there are obviously a number of issues with this approach, considering that it has to rely on some custom conversion of mongoDB operations to SQL and mimic the original internal data structure. Also concurrency is not fully supported, both because of the code and because SQLite can only handle a single process writing to the DB. It would likely require quite a lot of time to increase the quality and I cannot spend a lot of efforts on this.
However, I tested this version with a few trivial flows and it seems to work fine, so it could provide a very interesting alternative for newcomers or for running few workflows without the need of setting up a DB.

Now I am conflicted about if and how I should make this version available. My main concerns are:

• the whole solution is hackish and the code quality is not very high. I don't like very much the idea of inserting a big portion of code like this in a project like jobflow-remote, where we put a lot of efforts in keeping it clean.
• there will definitely be some bugs and one may get errors if trying to act on the DB while the runner is active. I am afraid that a new user will think that these shortcomings are intrinsic to jobflow-remote and decide to drop it altogether.
• Users may also get lazy and end up with large file based DBs, which is not optimal
• And of course this may increase the time that we should spend on maintaining this double data structure.

I have considered 3 options to distribute it, with some additional pros/cons on top of the points already mentioned.

1. insert it directly in the official jobflow-remote release with a JobController base class and Mongo/SQL subclasses:
   pros:

   • everybody can install, test it easily and then switch to the standard version
   • everything in the same place and possibility to access all the features

   cons:

   • In this view MongoDB and SQL seem on the same level, while it should be clear that the MongoDB implementation is the reference and preferred option.
   • needs to be maintained constantly together with the standard JobController
   • SQLJobController is in the official source code

2. push it to a specific branch in the main repo. Users install from source.
   pros:

   • stays tied to the original jobflow-remote
   • is not distributed as part of the official source code

   cons:

   • requires installing from source
   • more difficult for beginners
   • can still become outdated quickly

3. create a new package (e.g. jobflow-remote-file)
   pros:

   • can be easily installed
   • lives on its own and can be declared as a test project
   • all DB configurations in the project can be removed and have the file based as only option (quick setup)

   cons:

   • more difficult to keep it up to date with standard jobflow-remote (import every bugfix manually) and I am afraid the two projects may diverge over time.
   • more overhead in the maintenance of the package (another full repository to manage)
   • Even more likely that people will stick to this simplified version

I would really like to give this additional option, but all in all I am not really sure if it is worth to make the effort to release this and how. What do you think?

[Andrew-S-Rosen]

I think a lot of this depends on the code quality, which sounds like it is perhaps unknown. At the very least, you should push it to a public branch. Maybe you can label it experimental and link to it in the docs? Anyone curious can give it a try if they wish.

I guess the best long term solution would be to implement an SQL-based store in Maggma, right? Then no changes to JF Remote would be needed.

[JaGeo]

I agree with @Andrew-S-Rosen that a public branch would be very useful. Thus, 2.
Maybe some of use (including me) find some time to test it. Then, we can decide for 1 or 3.

[gpetretto]

Thank you for the feedback. I will clean it up a bit and push to a branch. I will maybe open a separate discussion about it, so that anyone interested can provide feedback and here we can keep for the general discussion about how to improve accessibility.

In the meanwhile I will maybe experiment with other solutions.

I guess the best long term solution would be to implement an SQL-based store in Maggma, right? Then no changes to JF Remote would be needed.

Unfortunately not, otherwise even the JsonStore or the MontyStore would have been enough.
In order to properly handle the concurrency, most of the operations on the DB need to be atomic. Plus we take advantage of aggregations probably other adanced features of mongodb. All these cannot be guaranteed by the API of the maggma Store, but are not even supported by mongomock or MontyDB. The problem is that it seems that it does not exist a file-based version of MongoDB remotely on par with SQLite.
Just to clarify, the queue database is defined as a maggma MongoStore just to have a single way of defining all the DBs in jobflow-remote (initially we had a separate input format for the queue and it was pointed out that was making things messy), but inside the JobController uses directly the db and collections.