First meeting of the machine learning working group

[nbren12]

cc @rabernat @jhamman @mrocklin @arokem @shoyer

Today, we spent some time in the weekly check-in meeting (#527 ) discussing how pangeo can support machine learning workflows. Overall, there is a lot of interest, and we just barely cracked the surface. To further discuss, we agreed to start a machine learning working group, and I think we should set up a video-call within the next couple of weeks. If you would like to participate, please fill in your availability on this doodle poll: https://doodle.com/poll/5f6x8dinmu5eetmn

I was not able to find github user names for everyone on today's call. Would someone be able to @ the people I missed (eg. Alando, Jim)?

As an agenda, we could build off the topics we mentioned today:

• Example workflows for climate ML - jupyter vs command line
• Data preprocessing and data loading
• Data structures (e.g. labeled arrays for deep learning, interoperability of numpy-like libraries)
• Infrastructure - GPUs, data storage
• Automation and reproducibility:
  • hyper-parameter tuning
  • workflow engines: snakemake, pachyderm
  • experiment loggers: sacred

As @mrocklin brought up, I think we should figure which of these items are general enough to lie within the scope of pangeo. Having a list of existing or envisioned machine learning workflows would help us do this. It seems that Anaconda+NASA has compiled a nice set of tutorials in the EarthML project. For my part, I recently gave an ML tutorial using binder.pangeo.io that went really smoothly.

What else do you all think we should discuss?

[alando46]

I like the idea of starting off with the workflows. It seems like it could be useful to try and coalesce on a common-ground workflow that is the ideal balance between flexibility/extendability and simplicity. Through that lens, we could also discuss what fits into the Pangeo scope.

Thoughts?

Looking forward to getting started.

[raspstephan]

Hi everyone.
I would also be interested in joining this (replied to the doodle already). I am working an a similar topic as Noah and am also struggling with large data amounts in my ML and analysis workflow for which I already use xarray. Maybe I can contribute some insights from a practitioner's perspective.

[gmaze]

Hi all,
I'm also very interested into this.
I primarily use ocean observations (in situ and satellite through xarray, eg: github.com/obidam/pyxpcm)
I found that a critical step is into the data preprocessing and loading, as it requires slow "stacking" of multiple dimensions and possibly interpolation along at least one of the dimension of the arrays.

[jhamman]

@nbren12 - was a date/time settled on here?

[RPrudden]

I'm also very interested, and will make the next meeting if I can.

[nbren12]

@jhamman I wanted to give people a few days to respond to the poll. I will select a date and time tonight.

Also, it seems like there is not a clear best option emerging, and there is no time where both @rabernat and @jhamman overlap :(. Is one of you particularly interested in attending?

[jhamman]

I've updated my availability. I can be available on the morning of the 25th.

[nbren12]

@jhamman Great! Sounds like we should go with that date.

Ok everyone. It looks like the best date for us to meet is Jan 25 from 10:00 AM to 11:00 AM (PST). I will follow up with details about the video conference.

I am sorry that @gmaze and Rachel cannot make that time. Hopefully, you can attend future calls.

[rabernat]

I have very limited availability during this period. It’s much more important that @RPrudden and @gmazd attend than me, given their expertise in this area. Please consider this when choosing the final time.

…

Sent from my iPhone

On Jan 14, 2019, at 10:43 PM, Noah D Brenowitz ***@***.***> wrote: @jhamman Great! Sounds like we should go with that date. Ok everyone. It looks like the best date for us to meet is Jan 25 from 10:00 AM to 11:00 AM (PST). I will follow up with details about the video conference. I am sorry that @gmaze and Rachel cannot make that time. Hopefully, you can attend future calls. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[nbren12]

Thanks for letting me know Ryan. Even with that in mind, I still think Jan 25 is the best day given everyone's availability.

Unfortunately, I think we are stuck in a local minimum for now. An alternative would by June 24th at 9am (PST), but then @shoyer, @alando46 , and @rabernat couldn't attend.

[nbren12]

Okay. I think we should finalize the date for Jan 25 from 10:00 AM to 11:00 AM (PST). Does that sounds okay? If so, thumbs up this post.

[rabernat]

@duncanwp is very interested in participating!

[djgagne]

I will be joining in as well.

[ktyle]

I would like to join in too!