Use Case Notebook for "Atmospheric Moisture Budgets"

[rabernat]

We need a preliminary notebook that uses Xarray to calculate something related to this use case. It doesn't have to be super long or complicated. It just has to express a real-world, scientifically relevant calculation on a large dataset.

Requirements:

• Carefully explain the science behind the calculation in words and equations.
• Clearly document the input dataset (preferably a standard product like ERA-interim or NCAR large ensemble)
• Use Xarray to load the dataset and, as much as possible, to make the calculations
• Where Xarray can't accomplish the necessary tasks, clearly document the other functions and / or outside packages that need to be employed
• If the analysis has been implemented in other languages (e.g. MATLAB, Ingrid, etc.), provide a link to the code and, if possible, the output (which can be used as validation)

cc: @naomi-henderson

I would like to use this as a template for the other three use cases, so I would appreciate general feedback on the checklist above. Are these the right requirements?

[rabernat]

We should add something about setting up the environment and enumerating the dependencies.

Here is my guide to setting up a conda environment for xarray work:
https://medium.com/@rabernat/custom-conda-environments-for-data-science-on-hpc-clusters-32d58c63aa95

But @darothen's "getting started" guide for gcpy is better and more comprehensive:
http://www.danielrothenberg.com/gcpy/getting_started.html

[rabernat]

You also asked about the gist extension:
https://github.com/mozilla/jupyter-notebook-gist

[darothen]

Please feel free to cannibalize my guide. If I can help in any way, do let me know.

[spencerahill]

Curious, what motivated this (and "the other three") use case? Just something that satisfies your above constraints?

I ask because I have worked on the problem of closing atmospheric column tracer budgets computed post-hoc. Depending on how concerned you are with closure, this can be pretty involved. The Appendices of our recent paper touch on this: http://journals.ametsoc.org/doi/abs/10.1175/JCLI-D-16-0785.1.

But I suspect this is more detail than you are looking for? If not, I can provide the Python functions that I defined for these efforts (they relied on my infinite-diff package and were computed via an aospy pipeline).

[rabernat]

@spencerahill: hopefully the project description I published earlier today clarifies what I meant by "use cases": these are the official use cases from the NSF proposal. However, we are certainly in need of as many use cases as possible and welcome contributions from everyone. If you have use cases you want to contribute, I would submit a PR to the pangeo-tutorials repo (currently empty, but hopefully will fill up with cool stuff!)

[rabernat]

Just a heads-up that the lack of any use case notebooks is now blocking progress on our project. It would be great to get some basic use cases checked in. They don't have to be fancy! Any sort of semi-realistic workflow on real data is enough for the systems people to move forward with analyzing performance.

[naomi-henderson]

The calculation of the eddy moisture transports is the most intensive I/O part of the budgets, so I am developing a preliminary notebook which computes the monthly mean co-variances from 6 hourly (e.g., u'q'). I will get the notebook working on cheyenne for the CMIP5 version of CESM. Once it is working, I will submit it to the group for critique before cleaning it up. Then I will start working on the vertical integrals, etc. Sound ok?

[mrocklin]

@naomi-henderson if you run into performance issues (like I/O bottlenecks) and want to try scaling things let me know. I'd be happy to try to collaborate to try to accelerate things.

[rabernat]

@naomi-henderson your plan sounds perfect

Also feel free to ping us for help with any xarray or general python questions you may encounter while developing your notebook. I know you're still spinning up on these tools, so we are here to help with that too.

[naomi-henderson]

@rabernat - I have made a preliminary version of an eddy covariance notebook and it works fine, albeit slowly, on my local machine (one year has to process 24G). I am having a few issues with the time grids, groupby, etc, as you can see here: covar notebook I may have wandered off into the deep end ...

[rabernat]

Awesome @naomi-henderson! This looks like a great start. I'm very pleased you have gotten your calculation to a preliminary working version.

I would love to sit with you for an hour on Wednesday and go though this in detail. It should be possible to eliminate the loops over files. Dask is unable to parallelize such operations, and they can also be confusing to read. The idea is to first create one big xarray dataset with everything in it (although not in memory) and then perform the desired grouping / aggregation on it using xarray's built in capabilities. You should never have to manually accumulate an average, as you do in your notebook--although this is a very natural place to start for someone coming from matlab or another lower level language!

I'm free all day on Wednesday, so please suggest a time that works for you!