Updated adpupa prepbufr api for the ObsError group variables #840
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…te obsfile for satwnd.abi_goes-16
Two updates for GOES AMV 1. BUFR Converter: - remove wind direction and speed - change sensorZenithAngle to satelliteZenithAngle 2. UFO Filters: - change `sensorZenithAngle` to `satelliteZenithAngle` - add `linear obs operator` section The declaration of `linear obs operator" is necessary for GOES AMV since there is no linearized component for applying the 10-meter factor. The increments look reasonable: ``` 0: ---------------------------------------------------------------------------------------------------- 0: Increment print | number of fields = 8 | cube sphere face size: C768 0: eastward_wind | Min:-1.642335e-01 Max:+1.660654e-01 RMS:+2.746685e-04 0: northward_wind | Min:-2.120068e-01 Max:+1.663539e-01 RMS:+2.631295e-04 0: air_temperature | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: surface_pressure | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: specific_humidity | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: cloud_liquid_ice | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: cloud_liquid_water | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: ozone_mass_mixing_ratio | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: ---------------------------------------------------------------------------------------------------- ``` The UFO filter results vs. GSI -- looks good  
Rather than each new ob type being manually added to the list, we can use `glob.glob` to get the list of available observation types.
Two updates for MetOp SCATWIND; one minor updates for GOES AMV
1. BUFR Converter:
- remove wind direction and speed
- add `stationElevation` as zero
2. UFO Filters:
- add linear obs operator section
The declaration of `linear obs operator" is necessary for GOES AMV since
there is no
linearized component for applying the 10-meter factor.
- add `Gaussian Thinning` filter to thin data with 75 km box (equal box
size, prefer data close to central of the box)
3. For GOES AMV YAML files, change output data extension from `nc4` to
`nc`
UFO vs. GSI O-F and QC comparison: O-F and data count passed look close
3Dvar increments are reasonable with MetOp-B SCATWIND alone.
```
0: ----------------------------------------------------------------------------------------------------
0: Increment print | number of fields = 8 | cube sphere face size: C768
0: eastward_wind | Min:-1.741092e+00 Max:+1.712287e+00 RMS:+2.282584e-03
0: northward_wind | Min:-1.825957e+00 Max:+1.930385e+00 RMS:+2.496075e-03
0: air_temperature | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
0: surface_pressure | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
0: specific_humidity | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
0: cloud_liquid_ice | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
0: cloud_liquid_water | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
0: ozone_mass_mixing_ratio | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
0: ----------------------------------------------------------------------------------------------------
```
3DVar increments are also reasonable with MetOp-B + MetOp-A assimilated
…oda.x (#730) This PR allows for all obtypes to be processed in two loops. We still need to figure out a way to process these in parallel (somewhat).
Using python multiprocessing to generate obs in parallel. The current list of obs goes from 10+ minutes to completing in ~5.5 minutes.
@RussTreadon-NOAA noted that `obs linear operator` should be `linear obs operator` in the GOES-17 satwind YAML. This fixes that.
This reduces runtime further to run everything in one parallel pool. Now, things complete in under 4 minutes (as of now).
Adding satwinds from the Advanced Himawari Imager (AHI) from Himawari-8 to GDASApp end-to-end testing new files include: parm/atm/obs/config/satwind_ahi_h8.yaml: QC filter YAML for AHI Himawari-8 satwinds parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_ahi.json: JSON containing data format, sensor, and satellite information for AHI Himawari-8 satwinds ush/ioda/bufr2ioda/bufr2ioda_satwind_amv_ahi.py: bufr2ioda code for extracting AHI Himawari-8 satwinds from BUFR modified files include: ush/ioda/bufr2ioda/run_bufr2ioda.py: added `"satwind_amv_ahi"` to list `BUFR_py` See #741 for details on testing. JEDI/GSI comparisons look good with GSI thinning turned off, but there are large ob-count disparities both before and after QC when comparing GSI+thinning to JEDI+thinning. The tested `Gaussian Thinning` filter is included in the YAML but commented out with a note. --------- Co-authored-by: Brett Hoover <bhoover@Orion-login-1.HPC.MsState.Edu> Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>
Instead of writing the filled out `.json` and `.yaml` files to `$COM_OBS`, we write them to `$DATA` so that they get purged unless `KEEPDATA="YES"`
Updating GNSSRO converter and adding the yaml file to GDASApp end-to-end testing based on [#712](#712). New file: parm/atm/obs/config/gnssro.yaml: Added QC filters in the YAML Updated files: parm/ioda/bufr2ioda/bufr2ioda_gnssro_bufr.json: Updated the satellite information ush/ioda/bufr2ioda/bufr2ioda_gnssro_bufr.py: Updated pccf, qfro, and satelliteAscending flag Refer to [issue #750](#750) for testing details. JEDI/GSI comparisons revealed a cutoff near obs=0.03 Rad due to super refraction 2. The number of observations after QC in JEDI is also inconsistent with GSI.
List of items to be added in this PR: - BUFR Converter: Add YAML for converting ATMS BUFR to IODA - The ObsValue is antenna temperature - Input BUFR is normal feel - Testing YAML: re-evaluation results due to update CRTM from v2.3.0 to v2.4.1-jedi.1 - atms_npp_noqc.yaml --- 100% replication - atms_npp.yaml --- 100 % replication - atms_n20_noqc.yaml --- 100 % replication - atms_n20.yaml --- ~ 1100% replication: two channel 7 observations difference - Config YAML: * add two read routine QC: data thinning and removal of data from scan edge - atms_npp.yaml - atms_n20.yaml - Paring PR in JCSDA UFO repository - This PR also has a [paring PR in JCSDA UFO repository](JCSDA-internal/ufo#3094). - It is OK to merge this one into GDASApp before the paring UFO PR - The related test data (obs and geoval files from GSI) for UFO Evaluation have been updated (due to CRTM-2.4.1 update) on ORION and HERA. Notes: - End-to-end testing completed without problem. - However, the filter results show one of the filters, 88-165 GHz scattering index check, tossed more data compared to GSI. - The excess screening from the 88-165 GHz scattering index check is under investigation and will be reported separately. - Please check comments below for validation results
…o-end testing (#764) Adding satwinds from the Spinning Enhanced Visible and InfraRed Imager (SEVIRI) from METEOSAT-8 to GDASApp end-to-end testing new files include: parm/atm/obs/config/satwind_seviri_m8.yaml: QC filter YAML for SEVIRI METEOSAT-8 satwinds parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_seviri.json: JSON containing data format, sensor, and satellite information for SEVIRI METEOSAT-8 satwinds ush/ioda/bufr2ioda/bufr2ioda_satwind_amv_seviri.py: bufr2ioda code for extracting SEVIRI METEOSAT-8 satwinds from BUFR End-to-end testing results are provided in #758 - Note: Thinning filter is provided in YAML but turned off due to irreconcilable differences with GSI in testing. --------- Co-authored-by: Brett Hoover <bhoover@Orion-login-1.HPC.MsState.Edu>
Small code-change to include SEVIRI/METEOSAT-11 satwinds in end-to-end validation. This uses the hook-ins already provided by the SEVIRI/METEOSAT-8 satwinds code that was merged in #764. New file: parm/atm/obs/config/satwind_seviri_m11.yaml - this is nearly a direct copy of parm/atm/obs/config/satwind_seviri_m8.yaml, since they utilize the same filters. Only the header information has changed. Modified file: parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_seviri.json - METEOSAT-11 was added to the `satellite_info` group End-to-end testing looks very similar to what was seen for SEVIRI/METEOSAT-8 satwinds, detailed in #758. A brief run-down: (LW)IR winds are in QC agreement down to a 0.02% difference (cloud-top)WV winds are in QC agreement down to a 0.20% difference VIS winds differ by 4.5%, but there are only 22 accepted winds in GSI and 23 in JEDI, the difference is a single satwind All ob, HofX, and ob-minus-HofX differences look good Increment summary, from gdasatmanlrun.log: ``` 0: ---------------------------------------------------------------------------------------------------- 0: Increment print | number of fields = 8 | cube sphere face size: C768 0: eastward_wind | Min:-3.479765e-01 Max:+3.430433e-01 RMS:+4.990993e-04 0: northward_wind | Min:-4.226316e-01 Max:+3.376561e-01 RMS:+4.889429e-04 0: air_temperature | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: surface_pressure | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: specific_humidity | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: cloud_liquid_ice | Min:+0.000000e+00 Max:+1.618770e-20 RMS:+1.293217e-23 0: cloud_liquid_water | Min:+0.000000e+00 Max:+1.474788e-19 RMS:+2.167418e-22 0: ozone_mass_mixing_ratio | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: ---------------------------------------------------------------------------------------------------- ``` Co-authored-by: Brett Hoover <bhoover@Orion-login-1.HPC.MsState.Edu>
In [ previous PR](#757) for ATMS, the end-to-end testing was done without data thinning. And the excessive QC filtering in clear-sky areas, compared to GSI, was found. (See [plots](#757 (comment))) This PR includes two fixes for ATMS in the end-to-end testing and one new feature: 1. fix the excess QC filtering in clear-sky areas. ---> This has a paring [UFO PR #3122](JCSDA-internal/ufo#3122) 2. add data thinning 3. add the diagnostic flags (QC) --- **This can [reproduce QC flags](#768 (comment)) from GSI in UFO** **To test updates in this PR, please check out the UFO branch: [feature/satrad](https://github.com/JCSDA-internal/ufo/tree/feature/satrad) from JCSDA-internal in gdas-validation. This branch consolidates all proposed code changes to UFO for gdas validation: UFO PR #JCSDA-internal/ufo#3122 UFO PR #JCSDA-internal/ufo#3121 UFO PR #JCSDA-internal/ufo#3094 --------- Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>
This PR includes the following: 1. YAML name change - add sensor (ABI) to the filename of observation YAML for SATWIND from GOES platforms 2. YAML name change - add sensor (ASCAT) to the filename of observation YAML for SCATWIND from MetOp platforms 3. update the YAML name changes in the list for 3dvar 4. Add `CO2` in obs forward operator and add `linear obs operator` without CO2 for ATMS These changes have no impact on 3DVar results.
This PR includes the following updates for conventional surface pressure 1. Add observation YAML under configuration 2. Update observation YAML under testing 3. Add conv_ps to 3dvar obs list 4. Set window shift to true (time boundary is inclusive) 5. Validate 3dvar for surface (station) pressure (end-to-end) - [QC validation](#803 (comment)) - [Increments](#803 (comment)) See [PR #792 ](#792) for the BUFR converter work to convert the following surface pressure observation into IODA: - SFC - 181, 187 - SFCSHIP - 180, 183 - SONDE - 120 The related UFO evaluation (using GSI obs and geovals) results for surface pressure are documented [here](https://docs.google.com/presentation/d/1OvVMYOutxVyk0GE93Jzoh3hQVoj-dCgrwsVWDDlPT1s/edit?usp=sharing).
PraveenKumar-NOAA
requested review from
RussTreadon-NOAA,
emilyhcliu,
ADCollard and
nicholasesposito
|
@emilyhcliu @ADCollard @CoryMartin-NOAA @RussTreadon-NOAA please review this PR, thanks. |
ADCollard
previously approved these changes
Jan 18, 2024
CoryMartin-NOAA
previously approved these changes
Jan 18, 2024
|
Do we want this in |
CoryMartin-NOAA
changed the base branch from
feature/gdas-validation
to
develop
CoryMartin-NOAA
dismissed stale reviews from ADCollard, nicholasesposito, and themself
The base branch was changed.
|
@PraveenKumar-NOAA now that I've rebased, can you resolve the conflicts? |
@CoryMartin-NOAA done, thanks. |
CoryMartin-NOAA
approved these changes
Jan 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR updates the adpupa prepbufr api for the ObsError group variables and also updates the function for the dateiTme variable.
The following file has been changed:
~/ush/ioda/bufr2ioda/bufr2ioda_adpupa_prepbufr.py