Possible regression with PyNIO data not being lazily loaded

[shoyer]

@WeatherGod reports on the mailing list:

I just tried [0.10.0 rc2] out in combination with the pynio engine (v1.5.0 from conda-forge), and doing a print on a dataset object causes all of the data to get loaded into memory.

[shoyer]

@WeatherGod can you verify that you don't get immediate loading when loading netCDF files, e.g., with scipy or netCDF4-python?

We did change how loading of data works with printing in this release (#1532), but if anything the changes should go the other way, to do less loading of data.

I'm having trouble debugging this locally because I can't seem to get a working version of pynio installed from conda-forge on OS X (running into various ABI incompatibility issues when I try this in a new conda environment).

[WeatherGod]

Is there a convenient sentinel I can check for loaded-ness? The only reason I noticed this was I was debugging another problem with my processing of HRRR files (~600mb each) and the memory usage shot up (did you know that `top` will report memory usage as fractions of terabytes when you get high enough?). I could test this with some smaller netcdf4 files if I could just loop through the variables and assert some sentinal.

…

On Thu, Nov 16, 2017 at 8:57 PM, Stephan Hoyer ***@***.***> wrote: @WeatherGod <https://github.com/weathergod> can you verify that you don't get immediate loading when loading netCDF files, e.g., with scipy or netCDF4-python? We did change how loading of data works with printing in this release ( #1532 <#1532>), but if anything the changes should go the other way, to do less loading of data. I'm having trouble debugging this locally because I can't seem to get a working version of pynio installed from conda-forge on OS X (running into various ABI incompatibility issues when I try this in a new conda environment). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1720 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARy-MO7la8KSJnQoto8Kso5gBYedUKQks5s3OgSgaJpZM4Qflk-> .

[shoyer]

The private Variable._in_memory attribute (e.g., ds.variables['temperature']._in_memory) will tell you if xarray thinks the data is in memory for the purposes of printing array values.

Are you using dask to load this data? That can also change things.

[fujiisoup]

I don't think we have tests of this case for any backend other than dask after #1532.
In my local tests, many backend array are loaded after print.

[fujiisoup]

OK. Found a bug and sent a PR.
This was the smallest bug fix I have ever done :)

[WeatherGod]

Awesome! Thanks!