Improve radar data acquisition #190

amotl · 2020-09-26T04:35:45Z

this implements #138 in order to also acquire radar data from /weather/radar besides /climate_environment/CDC/grids_germany/{5_minutes,hourly,daily}/radolan. It is based upon your #140.

After the refactorings coming from #188 and #189, I took the chance to rework your contribution coming from #140 based on these changes. It might save you quite some amount of work when picking this up again.

With kind regards,
Andreas.

codecov · 2020-09-26T04:43:48Z

Codecov Report

Merging #190 into master will increase coverage by 1.04%.
The diff coverage is 90.70%.

@@            Coverage Diff             @@
##           master     #190      +/-   ##
==========================================
+ Coverage   93.23%   94.28%   +1.04%     
==========================================
  Files          33       37       +4     
  Lines        1877     2116     +239     
  Branches      161      208      +47     
==========================================
+ Hits         1750     1995     +245     
- Misses         96       97       +1     
+ Partials       31       24       -7

Flag	Coverage Δ
#unittests	`94.28% <90.70%> (+1.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
wetterdienst/dwd/metadata/column_types.py	`100.00% <ø> (ø)`
wetterdienst/dwd/metadata/parameter.py	`100.00% <ø> (ø)`
wetterdienst/dwd/network.py	`100.00% <ø> (ø)`
wetterdienst/dwd/radar/sites.py	`44.06% <44.06%> (ø)`
wetterdienst/dwd/radar/api.py	`97.26% <97.26%> (ø)`
wetterdienst/dwd/radar/access.py	`98.18% <98.18%> (ø)`
wetterdienst/dwd/metadata/column_names.py	`100.00% <100.00%> (ø)`
wetterdienst/dwd/metadata/constants.py	`100.00% <100.00%> (ø)`
wetterdienst/dwd/metadata/time_resolution.py	`100.00% <100.00%> (ø)`
wetterdienst/dwd/observations/fields.py	`97.77% <100.00%> (-0.41%)`	⬇️
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7c618ed...9b52e65. Read the comment docs.

wetterdienst/dwd/radar/index.py

meteoDaniel · 2020-09-27T10:35:38Z

Dear @amotl thanks for working further on that issue. I had a lot of trouble the last two weeks.
did you take into account the changed data provision of the sweep data? It is now divided into two sub directories of simple and polarimetric filtered data.

amotl · 2020-09-27T20:52:49Z

Dear Daniel,

Did you take into account the changed data provision of the sweep data? It is now divided into two sub directories of simple and polarimetric filtered data.

I only recognized it and reflected it within

wetterdienst/tests/dwd/radar/test_index.py

Lines 123 to 138 in fc57ec3

    
           def test_radar_fileindex_sites_sweep_hdf5(): 
        
               file_index = _create_fileindex_radar( 
        
                   parameter=RadarParameter.SWEEP_VOL_VELOCITY_V, 
        
                   radar_site=RadarSites.BOO, 
        
                   radar_data_type=RadarDataType.HDF5, 
        
               ) 
        
               urls = file_index["FILENAME"].tolist() 
        
               assert any( 
        
                   "/weather/radar/sites/sweep_vol_v/boo/hdf5/filter_polarimetric" in url 
        
                   for url in urls 
        
               ) 
        
               assert any( 
        
                   "/weather/radar/sites/sweep_vol_v/boo/hdf5/filter_simple" in url for url in urls 
        
               )

Do you mean we should properly take that into account by adding yet another argument to the machinery, right? Like

class RadarFilterType(Enum):
    """ Enumeration for different Radar Filter Types"""

    SIMPLE = "simple"
    POLARIMETRIC = "polarimetric"

By introducing this argument, we will be able to properly address the artifacts on this level.

With kind regards,
Andreas.

wetterdienst/dwd/radar/metadata.py

amotl · 2020-09-27T22:11:15Z

Dear Daniel,

thanks for adding 810c7a1 in order to start bringing this to the top-level DWDRadarRequest API object.

22447d2 adds indexing support for all of:

25a2c64 finally implements accessing and downloading data from the locations at /weather/radar/{composit,radolan,sites} you and @kmuehlbauer have been aiming at within #138. Currently, only the respective -latest- files will be addressed. This might need some more polishing.

With kind regards,
Andreas.

So, what works for now is getting the -latest- files for other radar data beyond RADOLAN_GRID. Each one of these examples is representative for a different location on the DWD data repository (composit vs. radolan vs. sites).

Synopsis

# /weather/radar/composit
DWDRadarRequest(
    radar_parameter=RadarParameter.RX_REFLECTIVITY,
    date_times=RadarDate.LATEST.value,
)

# /weather/radar/radolan
DWDRadarRequest(
    radar_parameter=RadarParameter.RW_REFLECTIVITY,
    date_times=RadarDate.LATEST.value,
)

# /weather/radar/sites
DWDRadarRequest(
    radar_parameter=RadarParameter.DX_REFLECTIVITY,
    date_times=RadarDate.LATEST.value,
    radar_site=RadarSites.BOO,
)

Full examples

See also example/radar.

example/radar/radar_sweep_hdf5.py

wetterdienst/dwd/radar/access.py

kmuehlbauer · 2020-09-28T09:51:46Z

@amotl I'm currently testing this, but have some questions:

how to specifiy sweep_pcp_X data (single precipitation scan data, where X can be Z/V)
how to specify sweep_vol_X data (volume scan, where X can be Z/V)
how to request all elevations from sweep_vol data (00 to 09)

I'm trying to find my way through the package, but would need some guidance.

meteoDaniel

Thanks for this huge effort !

docs/pages/api.rst

wetterdienst/dwd/metadata/time_resolution.py

wetterdienst/dwd/radar/access.py

wetterdienst/dwd/radar/index.py

wetterdienst/dwd/radar/metadata.py

meteoDaniel · 2020-09-28T11:17:41Z

@amotl I'm currently testing this, but have some questions:

how to specifiy sweep_pcp_X data (single precipitation scan data, where X can be Z/V)

how to specify sweep_vol_X data (volume scan, where X can be Z/V)

how to request all elevations from sweep_vol data (00 to 09)

I'm trying to find my way through the package, but would need some guidance.

Dear @kmuehlbauer you can access data with:

    request = DWDRadarRequest(
        radar_parameter=RadarParameter.SWEEP_VOL_PRECIPITATION_V,
        date_times=RadarDate.LATEST.value,
        radar_site=RadarSite.BOO,
        radar_data_type=RadarDataType.HDF5,
    )

Afterwards you have to call request.collect_data . You have to pass an argument whether to store the file locally or not.

The mapping to the DWD abbreviations are:

    SWEEP_VOL_PRECIPITATION_V = "sweep_pcp_v"
    SWEEP_VOL_PRECIPITATION_Z = "sweep_pcp_z"
    SWEEP_VOL_VELOCITY_V = "sweep_vol_v"
    SWEEP_VOL_VELOCITY_Z = "sweep_vol_z"

But it could be possible that we will change this after your comment on that.

Elevations:
We are actually not covering the different elevations in sweep_vol. I think we have to introduce a new enum/parameter for such data.

amotl · 2020-09-28T11:49:00Z

How to obtain sweep_ data?

Right, thanks for answering already, @meteoDaniel. There's also an example program which outlines this, see

wetterdienst/example/radar/radar_sweep_hdf5.py

Lines 62 to 67 in db4fae2

    
           request = DWDRadarRequest( 
        
               radar_parameter=RadarParameter.SWEEP_VOL_PRECIPITATION_V, 
        
               date_times=RadarDate.LATEST.value, 
        
               radar_site=RadarSite.BOO, 
        
               radar_data_type=RadarDataType.HDF5, 
        
           )

kmuehlbauer · 2020-09-28T12:10:47Z

Right, thanks for answering already, @meteoDaniel. There's also an example program which outlines this

Thanks @amotl, I've found the example. What would I need to add for date_times (or another kwarg) to get the latest 10 files? I sense, that this might not be implemented yet.

amotl · 2020-09-28T12:14:42Z

What would I need to add for date_times (or another kwarg) to get the latest 10 files? I sense, that this might not be implemented yet.

That is true. No way to filter by "latest 10" for now.

I also believe this would be a way more convenient option than accessing by specific timestamps at all for this type of data, where there isn't really anything "historical" to access. Otherwise, one would have to figure out specific time ranges upfront, taking the respective intervals into consideration, etc. pepe.

kmuehlbauer · 2020-09-28T12:22:58Z

I also believe this would be a way more convenient option than accessing by specific timestamps at all for this type of data, where there isn't really anything "historical" to access.

True, the history is only three days IIRC. I would try to add code but I'm having a hard time to follow up through the abstraction levels.

amotl · 2020-09-28T12:29:04Z

I would try to add code but I'm having a hard time to follow up through the abstraction levels.

No worries, let me do that for you. I am about to switch over to a more OO-based approach, essentially mitigating the tedious passing of arguments around.

Thanks for all of your suggestions!

wetterdienst/dwd/radar/access.py

kmuehlbauer · 2020-10-09T07:25:48Z

This looks like it's almost finished. 🎉 Is there any timeline when this will hit master and possibly be released? Thanks!

amotl · 2020-10-09T11:21:02Z

Dear @kmuehlbauer,

This looks like it's almost finished. 🎉

After some adjustments I made to this yesterday, I believe it is ready now. Please advise if I missed to address anything.

Cheers,
Andreas.

kmuehlbauer · 2020-10-09T11:34:33Z

After some adjustments I made to this yesterday, I believe it is ready now. Please advise if I missed to address anything.

OK, this is great news, @amotl! I can't immediately see any problems so far. If there's something amiss, it can still be added later.

- Naming things - Improve indexing for directories containing both BIN and BUFR - More tests for "test_radar_fileindex"

- Add RadarParamters RW, RY and SF for RADOLAN - Add RadarParamter RADOLAN_GRID for addressing daily, hourly and 5_minutes resources - Swap FX against PP for composites - Add more tests covering the new options

amotl · 2020-10-09T17:29:21Z

Finally, this has been merged. Thanks for all of your support along the way, @meteoDaniel, @gutzbenj and @kmuehlbauer!

Is there any timeline when this will hit master and possibly be released? Thanks!

Will you cut a new release, @gutzbenj? Thanks already!

amotl requested review from meteoDaniel and gutzbenj September 26, 2020 04:35

amotl force-pushed the radar-more branch from 4fc20e4 to 2db77db Compare September 26, 2020 04:39

amotl mentioned this pull request Sep 26, 2020

WIP: Add more radar data #140

Closed

amotl changed the title ~~Acquire radar data from /weather/radar~~ Improve radar data acquisition Sep 26, 2020

amotl force-pushed the radar-more branch from 2db77db to 1b65244 Compare September 26, 2020 14:16

gutzbenj reviewed Sep 26, 2020

View reviewed changes

wetterdienst/dwd/radar/index.py Show resolved Hide resolved

gutzbenj approved these changes Sep 27, 2020

View reviewed changes

amotl commented Sep 27, 2020

View reviewed changes

wetterdienst/dwd/radar/metadata.py Show resolved Hide resolved

amotl commented Sep 27, 2020

View reviewed changes

wetterdienst/dwd/radar/metadata.py Outdated Show resolved Hide resolved

amotl force-pushed the radar-more branch 4 times, most recently from 3ca804a to df1f5ac Compare September 28, 2020 02:11

amotl commented Sep 28, 2020

View reviewed changes

example/radar/radar_sweep_hdf5.py Outdated Show resolved Hide resolved

amotl commented Sep 28, 2020

View reviewed changes

wetterdienst/dwd/radar/access.py Outdated Show resolved Hide resolved

amotl force-pushed the radar-more branch from 9d6180a to 5e9073d Compare September 28, 2020 05:41

meteoDaniel reviewed Sep 28, 2020

View reviewed changes

amotl force-pushed the radar-more branch from 2ca3920 to cf73cc8 Compare September 28, 2020 13:18

gutzbenj reviewed Oct 5, 2020

View reviewed changes

wetterdienst/dwd/radar/access.py Outdated Show resolved Hide resolved

amotl force-pushed the radar-more branch 4 times, most recently from dd294db to f4f5de6 Compare October 8, 2020 20:08

amotl force-pushed the radar-more branch from 84fa7cb to e7fc4d0 Compare October 9, 2020 12:50

d.lassahn and others added 15 commits October 9, 2020 14:52

Add enums and configs for radar data

0e68097

Radar data: Add functionality to file_index and file_path generation

200513a

Radar fixups

e9730f2

Improve radar data acquisition

3f6f199

- Naming things - Improve indexing for directories containing both BIN and BUFR - More tests for "test_radar_fileindex"

Start adding radar_data download into the top level API

361861a

Improve support for indexing more radar directories on opendata.dwd.de

0e5039e

- Add RadarParamters RW, RY and SF for RADOLAN - Add RadarParamter RADOLAN_GRID for addressing daily, hourly and 5_minutes resources - Swap FX against PP for composites - Add more tests covering the new options

Radar data: Implement acquisition of RADOLAN vs. non-RADOLAN data

c6be352

Fix MINUTES_10 typo again

e3fb3c2

Radar data: Remove PP_REFLECTIVITY

0d35755

Radar fixups

5e75ce6

Radar data: Fix tests

7efb203

Radar data: Add "elevation" argument for SWEEP_VOL_ data

5bd0984

Update radar site information

bea7332

Remove radar data store

03e3741

Update changelog

16f5ea1

amotl force-pushed the radar-more branch from e7fc4d0 to 16f5ea1 Compare October 9, 2020 12:53

amotl marked this pull request as ready for review October 9, 2020 12:53

Fix radar tests again

9b52e65

amotl merged commit 695222c into master Oct 9, 2020

amotl deleted the radar-more branch November 22, 2020 13:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve radar data acquisition #190

Improve radar data acquisition #190

amotl commented Sep 26, 2020 •

edited

Loading

codecov bot commented Sep 26, 2020 •

edited

Loading

meteoDaniel commented Sep 27, 2020

amotl commented Sep 27, 2020 •

edited

Loading

amotl commented Sep 27, 2020 •

edited

Loading

kmuehlbauer commented Sep 28, 2020

meteoDaniel left a comment

meteoDaniel commented Sep 28, 2020 •

edited

Loading

amotl commented Sep 28, 2020 •

edited

Loading

kmuehlbauer commented Sep 28, 2020

amotl commented Sep 28, 2020

kmuehlbauer commented Sep 28, 2020

amotl commented Sep 28, 2020 •

edited

Loading

kmuehlbauer commented Oct 9, 2020

amotl commented Oct 9, 2020

kmuehlbauer commented Oct 9, 2020

amotl commented Oct 9, 2020

Improve radar data acquisition #190

Improve radar data acquisition #190

Conversation

amotl commented Sep 26, 2020 • edited Loading

codecov bot commented Sep 26, 2020 • edited Loading

Codecov Report

meteoDaniel commented Sep 27, 2020

amotl commented Sep 27, 2020 • edited Loading

amotl commented Sep 27, 2020 • edited Loading

Synopsis

Full examples

kmuehlbauer commented Sep 28, 2020

meteoDaniel left a comment

Choose a reason for hiding this comment

meteoDaniel commented Sep 28, 2020 • edited Loading

amotl commented Sep 28, 2020 • edited Loading

kmuehlbauer commented Sep 28, 2020

amotl commented Sep 28, 2020

kmuehlbauer commented Sep 28, 2020

amotl commented Sep 28, 2020 • edited Loading

kmuehlbauer commented Oct 9, 2020

amotl commented Oct 9, 2020

kmuehlbauer commented Oct 9, 2020

amotl commented Oct 9, 2020

amotl commented Sep 26, 2020 •

edited

Loading

codecov bot commented Sep 26, 2020 •

edited

Loading

amotl commented Sep 27, 2020 •

edited

Loading

amotl commented Sep 27, 2020 •

edited

Loading

meteoDaniel commented Sep 28, 2020 •

edited

Loading

amotl commented Sep 28, 2020 •

edited

Loading

amotl commented Sep 28, 2020 •

edited

Loading