Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix for IGRA dewpoint calculation #227

Merged
merged 1 commit into from
Jul 9, 2018
Merged

Conversation

swnesbitt
Copy link
Contributor

There was an order of operations issue where the data being read dewpoint_depression was being subtracted from temperature before the field was flagged for missing values. This pull request simply changes the field being flagged and does the flagging before dewpoint is calculated, resulting in properly flagged dewpoints.

@CLAassistant
Copy link

CLAassistant commented May 3, 2018

CLA assistant check
All committers have signed the CLA.

@danielmwatkins
Copy link
Contributor

Thanks for catching that!

@dopplershift
Copy link
Member

Hmm...test failure due to lack of dewpoint_depression field. Is that only available sometimes?

@swnesbitt
Copy link
Contributor Author

swnesbitt commented May 4, 2018

All the humidity data is reported in dewpoint depression. This error only arises if there is a valid temperature and invalid dewpoint_depression observation at a level. I'm working with data from South America and this happens sometimes.

@dopplershift
Copy link
Member

Hmmm...is there any way to keep it from erring out in that case?

@swnesbitt
Copy link
Contributor Author

swnesbitt commented May 4, 2018 via email

@dopplershift
Copy link
Member

Let me clarify. With your changes, we are currently seeing an error in the tests:

            axis = self._get_axis_number(axis)
            agg_axis = 1 - axis
    
            agg_obj = self
            if subset is not None:
                ax = self._get_axis(agg_axis)
                indices = ax.get_indexer_for(subset)
                check = indices == -1
                if check.any():
>                   raise KeyError(list(np.compress(check, subset)))
E                   KeyError: ['dewpoint_depression']
../../../virtualenv/python3.6.3/lib/python3.6/site-packages/pandas/core/frame.py:3489: KeyError

So while maybe in theory your change shouldn't cause any errors, in practice it's causing an error for:

df, header = IGRAUpperAir.request_data(datetime(2014, 9, 10, 0),                           
                                            'USM00070026', derived=True)

@danielmwatkins
Copy link
Contributor

danielmwatkins commented May 4, 2018

The problem here is that this fix is only relevant to the derived=False case. When derived=True, the retrieved file will never have a dewpoint depression column. Instead it will have relative humidity. This is why the code as is has reported_relative_humidity in the QC step.

 df = df.dropna(subset=('temperature', 'reported_relative_humidity',
                        'u_wind', 'v_wind'), how='all').reset_index(drop=True)

So just take the change out of the `if self.suffix == '-drvd.txt' section, and leave the change in the 'else' section, and it should pass the tests.

@jrleeman
Copy link
Contributor

jrleeman commented Jul 5, 2018

Thanks @DanielWatkins and @swnesbitt - I just fixed this up a bit and rebased. Assuming that tests pass, we're good to go here.

@dopplershift dopplershift added this to the 0.8 milestone Jul 9, 2018
@dopplershift dopplershift merged commit 20c017a into Unidata:master Jul 9, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants