Pangeo kernel for EOX College

[annefou]

Working on providing EOX with a Pangeo image for the Pangeo exercises. We are now testing the new image on EOSC (Pangeo@EOSC). see PR: pangeo-data/pangeo-eosc#79

[annefou]

Some more details on the Pangeo kernel. We are using pangeo-notebook environment:
https://github.com/pangeo-data/pangeo-docker-images/tree/master/pangeo-notebook

The Docker image used and tested is:

  name: pangeo/pangeo-notebook
       tag: 2024.06.02

So to get the corresponding pangeo environment (conda lock, etc.), you need to fetch the corresponding tag:

https://github.com/pangeo-data/pangeo-docker-images/releases/tag/2024.06.02

[clausmichele]

@UndeadFairy could you start checking how to use the provided image in the EOxHub?

[UndeadFairy]

@clausmichele we discussed over the emails and solve it. We will discuss more on a meeting on Thursday.

[clausmichele]

New kernel name "pangeo" in EOxHUB. Dask Wdiget also included, it's necessary to test it.

[tinaok]

The Dask local cluster gets default IP address which is 127.0.0.1. That will not work when rendering it from dask lab extension.
We need whether to put configuration file in the $HOME/.config/dask
or set default enviroment variables.

Please refer the name of enviroment variable or configuration detail to
dask/distributed#7736
or
https://github.com/umr-lops/dask-hpcconfig/blob/9a8772f6fa9e905df056a4eb891590bfbc2b1230/dask_hpcconfig/processors.py#L10

[UndeadFairy]

We can set environmental variables for all users, that sounds like easiest option

[tinaok]

please add a file at ~/.config/dask/

08:40 $ cat distributed.yaml 
distributed:
  dashboard:
     link: "/user/{JUPYTERHUB_USER}/proxy/{port}/status"

[tinaok]

How many threads do we have? is it possible to have >2 so that here we can show the parallelisation?

[tinaok]

Kernel is good, but it would be nice to have more CPU and Memory resources. Is it possible to have 4 threads with 4 times the Memory while testing?

[Schpidi]

@tinaok we're currently using our free trial plan for the MOOC which provides only shared 4 GB memory and 1 CPU. Up to now this was sufficient but I agree for the dask exercise it would be good to have some more resources available. We'd be happy to configure more resources and are available to discuss the required configuration details like duration. We're happy to cover the incurred costs up to a limit like the free trials but depending on the requirements we should also start discussing funding options. Maybe ESA's NoR could be a mechanism?

[aapopescu]

@Schpidi, NoR could be an option. We should then consider a shared-resources cost model for course participants. Could you try to make an estimate of necessary resources per 100 participants?

[tinaok]

@Schpidi I think some testing with 4 threads with 4 times the Memory would be good to evaluate if this is not too much/ little, and wall_time required for learner to spend.
When can we make the test?

[Schpidi]

@tinaok apologies for the delay, maybe you've seen it already but there should now be a new profile available for your user for testing with more resources:

[annefou]

I tested the new profile with the pangeo kernel and it works. We can show the dask dashboard and users can learn how dask parallelisation "works". Now we need to work on the lesson itself to see if the resources are sufficient for a learner to get a good understanding of dask.

One thing that is still missing is the config file for dask. As mentioned earlier by @tinaok it makes it much easier for users. Each user needs to have a config file ~/.config/dask/distributed.yaml with:

distributed:
  dashboard:
     link: "/user/{JUPYTERHUB_USER}/proxy/{port}/status"

(see message from Tine above).

[Schpidi]

Added the environment variable DASK_DISTRIBUTED__DASHBOARD__LINK with value https://hub.eox.at/user/{USERNAME}/proxy/{port}/status directly in the workspace.