Approximate angles without TLE #65

ninahakansson · 2020-06-04T12:57:17Z

When a TLE close enough in time is missing use lat, lon of
mid-column for apprixmation of satellite position and Tiros-N height
from OSCAR as satellite altitude approximation.

When a TLE close enough in time is missing use lat, lon of mid-column for apprixmation of satellite position and Tiros-N height from OSCAR as satellite altitude approximation.

mraspaud

In some cases some of the parameters or angles are available. Should we try using these also when possible as a fallback when TLE is missing?

mraspaud

Interesting, thanks for submitting this! A couple of comments.

mraspaud · 2020-06-11T12:57:15Z

pygac/reader.py

+            from pyorbital.orbital import get_observer_look as get_observer_look_no_tle
+            sat_alt = 850.0  # km  TIROS-N OSCAR
+            sat_azi, sat_elev = get_observer_look_no_tle(
+                self.lons[:, 204][:, np.newaxis],
+                self.lats[:, 204][:, np.newaxis],  # approximate satellite position
+                sat_alt,  # approximate satellite altitude
+                self.times[:, np.newaxis],
+                self.lons, self.lats, 0)


Could you make this into a separate function?

mraspaud · 2020-06-11T12:58:05Z

pygac/reader.py

+            from pyorbital.orbital import get_observer_look as get_observer_look_no_tle
+            sat_alt = 850.0  # km  TIROS-N OSCAR
+            sat_azi, sat_elev = get_observer_look_no_tle(
+                self.lons[:, 204][:, np.newaxis],
+                self.lats[:, 204][:, np.newaxis],  # approximate satellite position
+                sat_alt,  # approximate satellite altitude
+                self.times[:, np.newaxis],
+                self.lons, self.lats, 0)


Should we take into account the static attitude shifts from some satellites for this?

I do not know what the static attitude shifts is. Something included in the data? And I guess the sat_alt should be different for different satellites too.

sometimes it is in the data (for klm), otherwise we have these:

pygac/pygac/reader.py

Lines 51 to 75 in 3162edd

rpy_coeffs = {

'noaa7': {'roll': 0.000,

'pitch': 0.000,

'yaw': 0.000,

},

'noaa9': {'roll': 0.000,

'pitch': 0.0025,

'yaw': 0.000,

},

'noaa10': {'roll': 0.000,

'pitch': 0.000,

'yaw': 0.000,

},

'noaa11': {'roll': -0.0019,

'pitch': -0.0037,

'yaw': 0.000,

},

'noaa12': {'roll': 0.000,

'pitch': 0.000,

'yaw': 0.000,

},

'noaa14': {'roll': 0.000,

'pitch': 0.000,

'yaw': 0.000,

}}

ghost · 2020-09-09T15:52:35Z

Congratulations 🎉. DeepCode analyzed your code in 4.614 seconds and we found no issues. Enjoy a moment of no bugs ☀️.

👉 View analysis in DeepCode’s Dashboard | Configure the bot

👉 The DeepCode service and API will be deprecated in August, 2021. Here is the information how to migrate. Thank you for using DeepCode 🙏 ❤️ !

If you are using our plugins, you might be interested in their successors: Snyk's JetBrains plugin and Snyk's VS Code plugin.

…gac into get_angles_without_tle

carloshorn · 2021-01-14T15:09:21Z

Hi @ninahakansson,
we used this PR to process the entire TIROS-N data and compare it with the one using TLEs.
Unfortunately, it still requires some work, because we could not use the output to generate AMVs due to missing angles.
It appears like the algorithm that computes the sensor angles produces NaNs in the output.

See the central NaN stripe in the above image taken from the produced file AVHRR-GAC_FDR_1C_TSN_19790201T020219Z_19790201T035622Z_R_O_20201008T100626Z_0100.nc.

carloshorn · 2021-01-14T15:24:15Z

We also looked into potential other satellites that could benefit from this method, and it turned out that NOAA7 also shows large TLE gaps.

In order to avoid a static constant altitude for each satellite, I would propose to interpolate the satellite altitude from surrounding TLEs, because we can expect that the changes in altitude are smooth, except there was a manoeuvre. @mraspaud, is pyorbital capable to do this?

As @mraspaud proposed, we may also have a look into the angles inside the file and compare them to the fix presented in this PR.

ninahakansson · 2021-01-14T15:43:47Z

@carloshorn I checked and I get the same error. For some pixels in column 204 there are nodata instead of 0. I guess we could just put it to zero.

ninahakansson · 2021-01-14T15:57:15Z

I pushed a fix for the NaN for satzenith angle. Looking at the code I suspect the problem is really in pyorbital.

As for using the correction for noaa7 the height (according to OSCAR) is for NOAA-7 850km that is quite close to the 860km for TIROS-N. I think as a first approximation the fixed value of 860km is Ok. Interpolation using existing TLEs could be added later in an other PR. But I think it is good if we keep a default value as this means pygac can run also when TLEs are completely missing.

mraspaud · 2021-01-14T19:33:39Z

@carloshorn Pyorbital can't do interpolation of the TLEs, only propagation (extrapolation).

carloshorn · 2021-01-15T14:33:03Z

I see the advantage of having a TLE independent fall back mechanism.
For the FDR generation, I would prefer an algorithm that makes use of the entire TLE record, treating each TLE as an observation to determine the actual flight path.
@mraspaud, do you know if there are any libraries that we could use? Or do you know any algorithms in this direction that we may include into PyTroll? I could ask some people working on space debris at ESOC. Maybe they have an idea about the most accurate way to bridge TLE gaps.
Quick paper search gave me https://arxiv.org/pdf/1002.2277.pdf

mraspaud · 2021-01-15T14:55:41Z

@carloshorn I don't know of any library for doing TLE interpolation, but I haven't looked in a while. The fact that I'm no flight dynamics expert doesn't help, but I guess you have these people in house, right?
The paper you show seems promising, and that method would be a great addition to pyorbital :) 👍

ninahakansson · 2021-01-15T15:37:36Z

@carloshorn I think it is a good idea to include more accurate height estimates. But I will not have time to include this any time soon.

carloshorn · 2021-01-26T12:57:11Z

pygac/reader.py

+            self.times[:, np.newaxis],
+            self.lons, self.lats, 0)
+        # Sometimes the get_observer_look_not_tle returns nodata instead of 90.
+        sat_elev[:, mid_column] = 90


Since pyorbital is part of the PyTroll libraries, I would prefer to fix the issue there instead of fixing the library output in pygac. The issue is very likely a problem with the defined domain of trigonometric functions and their inverse, my candidates are arctan and arcsin.

Absolutely! But with the fix here it is ok to use also an old version of pyorbital. I think that is nice. I have made an issue to pyorbital pytroll/pyorbital#72.

I was thinking whether that correction could be applied only if the pyorbital version <= current version. This would remind us to remove that fix in the future. On the other hand we don't know whether the issue will be solved in the next pyorbital release. So what about referencing the github issue in the comment?

carloshorn · 2021-02-18T15:40:17Z

An additional note on using this fix together with pygac-fdr.
It turned out, that the IndexError in Reader.get_tle_lines will also hit the metadata collector.
In my quick and dirty test, I avoided this by returning empty strings instead of raising the index error, but raising the exception would be cleaner.
This is just a reminder for opening a ticket on pygac-fdr in case this fix produces useful data and gets merged into master.

sfinkens

Nice work! Just one minor comment and question.

sfinkens · 2021-03-16T08:07:19Z

pygac/reader.py

+            self.times[:, np.newaxis],
+            self.lons, self.lats, 0)
+        # Sometimes the get_observer_look_not_tle returns nodata instead of 90.
+        sat_elev[:, mid_column] = 90


I was thinking whether that correction could be applied only if the pyorbital version <= current version. This would remind us to remove that fix in the future. On the other hand we don't know whether the issue will be solved in the next pyorbital release. So what about referencing the github issue in the comment?

sfinkens · 2021-03-16T08:10:27Z

pygac/tests/test_reader.py

+        expected_sat_azi_0 = np.array([283.09872924, 283.12775589, 283.13951497, 283.14786413, 283.19638805])
+        expected_sat_azi_201 = np.array([272.85051989, 273.79847634, 272.04794616, 273.01363377, 274.00055397])
+        expected_sat_azi_408 = np.array([39.77021472, 39.71516966, 39.68104134, 39.60503726, 39.5403431])
+        expected_sat_elev_0 = np.array([20.94889204, 20.96041284, 20.96368521, 20.96826309, 20.95849212])
+        expected_sat_elev_204 = np.array([89.24533884, 89.22663677, 89.25079817, 89.24938043, 89.23004118])


Were these expected values computed with TLE and then for the tests they are approximated without TLE?

Yes, that is how I made theml.

mraspaud

Thanks for improving this, is there a test to update or add for this?

mraspaud · 2021-05-26T19:41:40Z

pygac/reader.py

+        # Sometimes (pyorbital <= 1.6.1) the get_observer_look_not_tle returns nodata instead of 90.
+        # Problem solved with https://github.com/pytroll/pyorbital/pull/77
+        if LooseVersion(pyorbital.__version__) <= LooseVersion('1.6.1'):
+            sat_elev[:, mid_column] = 90


Could you issue a warning here saying eg that for best results, pyorbital > 1.6.1 is needed?

Results are not better with pyorbital > 1.6.1. With the fix they are the same (= 90). Without the fix they sometimes get rounded to 89.999999 instead of 90 but I don't think that is better. I added a test to check sat_elev at nadir are 90 degrees.

Signed-off-by: Nina.Hakansson <a001865@c21515.ad.smhi.se>

mraspaud · 2021-06-18T09:08:11Z

pygac/reader.py

@@ -739,6 +740,7 @@ def get_sat_angles_without_tle(self):
        # Sometimes (pyorbital <= 1.6.1) the get_observer_look_not_tle returns nodata instead of 90.
        # Problem solved with https://github.com/pytroll/pyorbital/pull/77
        if LooseVersion(pyorbital.__version__) <= LooseVersion('1.6.1'):
+            pass


Why do you need a pass here?

mraspaud

Thanks for fixing this, LGTM

mraspaud · 2021-06-18T10:11:19Z

I made a mess here, I'll fix it after lunch

codecov · 2021-06-18T20:13:06Z

Codecov Report

Merging #65 (c53b737) into main (b4a9731) will increase coverage by 0.36%.
The diff coverage is 95.00%.

@@            Coverage Diff             @@
##             main      #65      +/-   ##
==========================================
+ Coverage   66.30%   66.66%   +0.36%     
==========================================
  Files          34       34              
  Lines        2858     2895      +37     
==========================================
+ Hits         1895     1930      +35     
- Misses        963      965       +2

Impacted Files	Coverage Δ
pygac/reader.py	`63.79% <88.23%> (+0.68%)`	⬆️
pygac/tests/test_reader.py	`99.32% <100.00%> (+0.05%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b4a9731...c53b737. Read the comment docs.

Approximate angles without tle

0b32a88

When a TLE close enough in time is missing use lat, lon of mid-column for apprixmation of satellite position and Tiros-N height from OSCAR as satellite altitude approximation.

mraspaud reviewed Jun 6, 2020

View reviewed changes

mraspaud reviewed Jun 11, 2020

View reviewed changes

Nina.Hakansson added 3 commits June 25, 2020 14:19

Created function for satellite angles without TLEs

d6b59fe

Do not assume 409 columns

8294e90

Test the get angles without tle

a650a16

Fixing style errors.

6346817

ninahakansson marked this pull request as ready for review September 21, 2020 12:30

Nina.Hakansson added 2 commits November 2, 2020 12:39

style (codefactor)

22a1c79

Merge branch 'get_angles_without_tle' of github.com:/ninahakansson/py…

55dec03

…gac into get_angles_without_tle

Fix problem at nadir

b3b535d

carloshorn reviewed Jan 26, 2021

View reviewed changes

sfinkens reviewed Mar 16, 2021

View reviewed changes

Nina.Hakansson added 2 commits May 26, 2021 10:11

Only do fix for older versions of pyorbital

efaa8e0

Added comment about pyorbital issue

b273b0a

mraspaud reviewed May 26, 2021

View reviewed changes

Nina.Hakansson added 3 commits May 27, 2021 13:40

Added test that trigger pyorbital <= 1.6.1 bug

46c5c36

Expanded and renamed test of nadir sat elevation

00d5bca

Added a warning about satellite height which is approximated to 850km

676a4c2

Signed-off-by: Nina.Hakansson <a001865@c21515.ad.smhi.se>

mraspaud reviewed Jun 18, 2021

View reviewed changes

Remove pass introduced by mistake

23c1e0f

mraspaud approved these changes Jun 18, 2021

View reviewed changes

mraspaud force-pushed the get_angles_without_tle branch from ebb37a8 to 23c1e0f Compare June 18, 2021 10:50

mraspaud and others added 3 commits June 18, 2021 12:58

Merge branch 'main' into get_angles_without_tle

9f5c66a

Fixing the angle tests

f08178f

Fix one more reader_test broken at merge

c53b737

mraspaud merged commit 5423880 into pytroll:main Jun 18, 2021

mraspaud mentioned this pull request Oct 25, 2022

Make angles computation optional for KLM #63

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Approximate angles without TLE #65

Approximate angles without TLE #65

ninahakansson commented Jun 4, 2020

mraspaud left a comment

mraspaud left a comment

mraspaud Jun 11, 2020

mraspaud Jun 11, 2020

ninahakansson Jun 11, 2020

mraspaud Jun 11, 2020

ghost commented Sep 9, 2020 •

edited by ghost

Loading

carloshorn commented Jan 14, 2021

carloshorn commented Jan 14, 2021

ninahakansson commented Jan 14, 2021

ninahakansson commented Jan 14, 2021 •

edited

Loading

mraspaud commented Jan 14, 2021

carloshorn commented Jan 15, 2021

mraspaud commented Jan 15, 2021

ninahakansson commented Jan 15, 2021

carloshorn Jan 26, 2021

ninahakansson Jan 26, 2021

sfinkens Mar 16, 2021

carloshorn commented Feb 18, 2021

sfinkens left a comment

sfinkens Mar 16, 2021

sfinkens Mar 16, 2021

ninahakansson Mar 16, 2021

mraspaud left a comment

mraspaud May 26, 2021

ninahakansson May 27, 2021

mraspaud Jun 18, 2021

mraspaud left a comment

mraspaud commented Jun 18, 2021

codecov bot commented Jun 18, 2021 •

edited

Loading

	rpy_coeffs = {
	'noaa7': {'roll': 0.000,
	'pitch': 0.000,
	'yaw': 0.000,
	},
	'noaa9': {'roll': 0.000,
	'pitch': 0.0025,
	'yaw': 0.000,
	},
	'noaa10': {'roll': 0.000,
	'pitch': 0.000,
	'yaw': 0.000,
	},
	'noaa11': {'roll': -0.0019,
	'pitch': -0.0037,
	'yaw': 0.000,
	},
	'noaa12': {'roll': 0.000,
	'pitch': 0.000,
	'yaw': 0.000,
	},
	'noaa14': {'roll': 0.000,
	'pitch': 0.000,
	'yaw': 0.000,
	}}

Approximate angles without TLE #65

Approximate angles without TLE #65

Conversation

ninahakansson commented Jun 4, 2020

mraspaud left a comment

Choose a reason for hiding this comment

mraspaud left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented Sep 9, 2020 • edited by ghost Loading

👉 View analysis in DeepCode’s Dashboard | Configure the bot

👉 The DeepCode service and API will be deprecated in August, 2021. Here is the information how to migrate. Thank you for using DeepCode 🙏 ❤️ !

carloshorn commented Jan 14, 2021

carloshorn commented Jan 14, 2021

ninahakansson commented Jan 14, 2021

ninahakansson commented Jan 14, 2021 • edited Loading

mraspaud commented Jan 14, 2021

carloshorn commented Jan 15, 2021

mraspaud commented Jan 15, 2021

ninahakansson commented Jan 15, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

carloshorn commented Feb 18, 2021

sfinkens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mraspaud left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mraspaud left a comment

Choose a reason for hiding this comment

mraspaud commented Jun 18, 2021

codecov bot commented Jun 18, 2021 • edited Loading

Codecov Report

ghost commented Sep 9, 2020 •

edited by ghost

Loading

ninahakansson commented Jan 14, 2021 •

edited

Loading

codecov bot commented Jun 18, 2021 •

edited

Loading