Running mesa in the cloud.

[smubita]

This discussion goes along with another topic here.

An option could be added to mesa to allow the model to be run in the cloud. This would be helpful for people with limited laptops available to them, or to people with really large datasets. Or people who are not able to run long jobs locally for whatever reason.

The command could look something like

mesa runserver --cloud=gcp

A prerequisite would be setting up your account for the cloud service in question, and authenticating to it from your local machine. And then installing the appropriate command line tools. In the case of GCP, you'd install the gcloud cli program, authenticate and set up your environment.

Mesa would then build a docker image using the existing image file (after having copied the model files into the image as the Dockerfile currently does with wolf_sheep ).

We could pick a particular way of running that image. If we used the batch running facilities available from GCP or AWS, the person running the model would have to push the image up to a repository they have access it to, and then they could run it from there.

Rather than including dependencies for two or three different cloud in the mesa project, we would run the appropriate cloud command ( gcloud or awscli ). mesa could assume the appropriate executable is in the path. Alternatively, mesa could just display the appropriate command to be copied and pasted.

Of course, if the person was comfortable running a container in the cloud, they could take care of all of the above. But it would be nice to make the process as easy as possible.

[houssam7737]

Talked to @jackiekazil about this, a couple more thoughts to add (everything you said here makes sense to me btw):

• Given that mesa is used by many academics, we might want to support/look into SLURM since I know many academics have access to a university supercomputer cluster instead of cloud products, and those are usually managed by SLURM as the job scheduler.
• We might want to look into a mutli-cloud tool instead of having to write code for GCP, AWS and Azure separately

Let me know what you think

[jackiekazil]

/cc @rht, since it looks like github didn't cross link to discussions.

[houssam7737]

As an update, preliminary research tells me using Ray or a similar framework would make this pretty neat and generalizable. We don't need to reinvent the wheel here, and I can add a super clear Readme on how to set this up with different cloud solutions.

[rht]

I agree with standing on the shoulder of Ray. This tutorial can be repurposed to execute Mesa code. And the steps could be encapsulated in a script, that is then called from mesa runserver.

[smubita]

I like both of your ideas. I am not familiar with either SLURM or any of the multitool frameworks, but would be glad to learn. @houssam7737 , perhaps you could post your Readme here as a good starting point?

[smubita]

This is what you're referring to, right? https://docs.ray.io/en/latest/index.html

[smubita]

I'm starting to look at Ray, although I'm not really familiar with it. @houssam7737 , would you like to chat about this?