-
Notifications
You must be signed in to change notification settings - Fork 23
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix ek80 calibration #56
base: RHT-EK80
Are you sure you want to change the base?
Conversation
Hi Ibrahim, Can you confirm that this issue is from the 2021 data you are working with. Specifically the data recorded using EK80 v2.1.x? I believe that this issue is a result of a file format change made by Simrad in a recent EK80 version. When converting FM data recorded on a system without an FM calibration applied one must look up gain and sa correction values from the gain and sa correction tables present in the configuration header. These tables are indexed by pulse duration, so one first finds the index of the pulse duration in the pulse duration table, then uses that index to extract the proper gain and sa correction. Within the configuration header there are two pulse duration tables: pulse_duration and pulse_duration_fm. The index for gain and sa correction for CW data is obtained from the pulse_duration. You would think it would be safe to assume that the index for FM data would be obtained from pulse_duration_fm, but up until this latest version that was not correct and pyEcholab always looked up the index from pulse_duration regardless of FM or CW operation. It seems that this has changed with files recorded using EK80 version > 2.1.x. The error you are running into is a result of the lookup failing in the pulse_duration. The latest commit has a fix that looks up the index from pulse_duration_fm if it fails to find a match in pulse_duration. I need to do additional testing, and I would like to confirm my theory re: change with v2.1.x, but I think this resolves the issue and allows for backwards compatibility with older files. I will look into this further and comment here. |
Sorry for the late reply, I was on a leave earlier. This issue is not only from processing the 2021 data, but from the 2020 data from our Sandeel survey. Where can I check the version of the EK80 data? I'll try to run with the latest update to check if the problem has been resolved. Thank you! |
I can confirm that the latest commit still produces error during calibration. However, this doesn't happen anymore with the 2021 data, only with the 2020 EK80 data: File "CRIMAC_preprocess.py", line 217, in process_data_to_xr
cal_obj = raw_data.get_calibration()
File "/home/a5414/.local/lib/python3.8/site-packages/echolab2/instruments/EK80.py", line 2422, in get_calibration
cal_obj.from_raw_data(self)
File "/home/a5414/.local/lib/python3.8/site-packages/echolab2/instruments/EK80.py", line 4243, in from_raw_data
super(ek80_calibration, self).from_raw_data(raw_data,
File "/home/a5414/.local/lib/python3.8/site-packages/echolab2/instruments/util/simrad_calibration.py", line 259, in from_raw_data
self.set_attribute_from_raw(raw_data, param_name,
File "/home/a5414/.local/lib/python3.8/site-packages/echolab2/instruments/util/simrad_calibration.py", line 371, in set_attribute_from_raw
param_data = self.get_attribute_from_raw(raw_data, param_name,
File "/home/a5414/.local/lib/python3.8/site-packages/echolab2/instruments/EK80.py", line 4359, in get_attribute_from_raw
gain = param_data[match_idx][0]
IndexError: index 0 is out of bounds for axis 0 with size 0 You can find the sample data in the Azure share path: Thank you. |
I looked into this further and realize that I need additional information from Simrad to determine the correct fix.
The issue here is that the files in question do not have a matching pulse duration for the commanded 2048 us in the I will attempt to get this info from Simrad, but they have been extremely reluctant to provide any information regarding their file format beyond what is published in the EK80 interface spec doc which does not include details such as this. I will also contact Echoview to see if they have any additional information on this. My guess is that the lookup for FM data without a BB cal should come from |
Thank you @rhtowler ! We have asked Simrad to look into it. I'll include @nilsolav and @tomaszfurmanek here for future follow up on this issue. |
I have not received any further guidance from Simrad but have been told that they will be releasing a reference implementation of FM to Sv conversion "soon" where this should be answered. I am anticipating that my original implementation where gain for FM data without a wideband calibration is extracted from the |
We have some EK80 files that triggered errors when calling
get_calibration()
. I'm quoting Tomasz's email below:It seems that
param_data[match_idx]
can be empty at times. This fix can avoid the error, however I'm unsure if this is the correct way to do it.Thanks!