Create epochs on-the-fly when loading data if data has no previous epoch information

[MichaelHuth]

• Check through all provided data from Think about adding/fixing up labnotebook entries on NWB export #881 and design workarounds for missing data if possible
• find missing data if possible in other files (t-b)
• Move epoch on demand recreation to EP_FetchEpochs
• Add caching for recreated epochs

Idea for missing custom waves

Some files have missing custom waves for stimset recreation.

If these custom waves stimset epochs are neighboured by square epochs with amplitude 0 (check in stimset note) and assusimg the custom waves have a non-zero amplitude, these can be found in the DA wave.
From that the length can be determined and the stimset note updated.
This can be done for each custom wave from left to right until the length of all stimset epochs is present.
(How precise the intermediate backward transformation from data wave to stimset indices is, has to be determined)

Idea for missing TP information on very early MIES data

Resample recreated stimset to a temp data wave and match that with existing DA wave.
Determine TP + onsetUser as well as termination delay from there.

close #2008

requires #2015

[t-b]

Allen FTP: epochs_issue_240308 has stimProtocol_2015_05_26_124639.h5 which holds the build stimsets. I think we could extract the custom waves from these.

[MichaelHuth]

Issues:

WB Formulas in nwb2_H17.03.016.11.09.01.nwb

Determine if insert TP was on in sweep 0
nwb2_Pvalb-IRES-Cre_Ai14-214784.03.02.nwb

[MichaelHuth]

@t-b How should the new feature coupled to the rest of MIES?

I currently have the fallback mechanism in BSP for displaying epochs in the SB/DB and the AB function to load Sweeps and Stimsets. Should there be a button for that in the AB or do we replace a button in the AB with loading both?

Also, what should be the update mechanism for the LBN and when do we apply that?

[t-b]

I've chatted with Tim about this and here is our conclusion:

• Overwrite the existing epoch info in NWBv2 files, existing IPNWB functions: WriteSingleChannel, WriteEpochs, AppendEpochTable
• Should be done only on request (i.e. just implement a function)
• We will then do the epoch overwriting with an approach like MIES_MassExperimentProcessing.ipf

We can do that in another PR given that this PR is self-contained with tests.

[t-b]

We also would like to have retrofitted epoch info for data loaded into the databrowser/sweepbrowser (aka on-the-fly creation) either for SweepFormula or for display. Should probably be cached for speed reasons. But this should only be done if we don't have any epoch info at all.

[t-b]

Thanks for the massive effort:

Review:

809d902 (WB: Better handling of errors when building a stimset, 2024-03-08)

• The key added to the wave note needs to be more specific. Something like "WaveBuilder Error"?
• Please introduce a constant for the error code 1. This makes it better discoverable (aka searching works better)
• Please also add a test or two for the new error handling. UTF_WaveBuilder.ipf seems fitting.
• Needs to raise STIMSET_NOTE_VERSION and adapted documentation in WB_GetWaveNoteEntry

cef5c04 (WB: Do not ASSERT in WB_StimsetHasLatestNoteVersion for non-existing stimset, 2024-03-21)

Makes sense

f51fc94 (WB: Transfer notes of stimset sweeps earlier in WB_GetStimset, 2024-03-21)

• Should also be covered in the tests.

71eb1b0 (EP: Add error handling if certain required values are missing, 2024-03-08)

• You seem to be working around an IP error with the "" + s.XXX. Please
  add a @todo workaround ... comment with a reference like #XXXX to the
  reported WM issue.

• Why can't you return early in EP_GetStimEpochsOffsetAndLength instead of setting err = 1?

• Please add an empty comment line between @param and @retval in EP_AddEpochsFromStimSetNote for visual separation

e0472f7 (DC: Recreation workaround for missing samplingInterval and baselineFrac, 2024-03-08)

• Assertion for empty DACList

DC_RecreateDataConfigurationResultFromLNB_baselineFrac_Path2:

• The level of 1E-3 seems to be arbitrary. As you are looking at the DA
  wave, can't we just a level of GetMachineEpsilon? I'm using that in the
  analysis functions as well.

• Commented out code
  // s.baselineFrac ...

• Who is responsible for setting

s.baselineFrac and friends to NaN if the following case is hit?

+    if(IsNaN(startIndex) || IsNaN(endIndex))
+        return NaN
+    endif

e4e82b7 (DC: Evaluate if the recreated stimset had an error condition, 2024-03-08)

Makes sense.

c0fb94c (DC: Better error messages for invalid stimsets after recreation by WB, 2024-03-12)

Nice!

eb56bab (DC: More flexible TP finding as fallback, 2024-03-12)

• That should probably be part of e0472f7 or?

24043f3 (DC: Add fallbacks in recreation for MIES version < 0.7, 2024-03-20)

• Commit message: "its necessary" -> "if necessary" ?

• While still correct it is confusing to see mean being used in
  DC_FindStimsetOffset as we can also use V_avg from the earlier WaveStats call

4faf20c (DC: Add fallback for DataConfigurationResult recreation if no stimset wave is present, 2024-03-27)

Makes sense.

e9612a8 (AB: Introduce button to load all stimsets and sweeps for the selected entries, 2024-03-08)

• You are duplicating code in AB_LoadStimsetsAndSweeps from
  AB_ButtonProc_LoadSweeps? Why not just PGC to both the stimset and the sweep
  buttons? Alternatively factor the code out into a common function.

3ba1af6 (EP: Change ASSERT to error return in EP_AddEpochsFromStimSet, 2024-03-12)

Fine!

c9c92e1 (Integrate epoch recreation in MIES, 2024-03-08)

I'm not particular fond of the TS/nonTS variants of EP_FetchEpochs but I also don't see another way.

• Please add documentation for BSP_GetSweepDF and move the introduction into its own commit.

• BSP_GetSweepDF: I don't see why we need the second if(isSweepBrowser) clause, I think this can be moved into the first one.

• BSP_GetSweepDF: Don't use SFH_ASSERT outside of SF. You can return $"" if you want to handle the non-bound case gracefully.

• EP_FetchEpochsFromRecreation. You need to return early if the returned epochs wave is null

• Can we exercise the code paths for on-the-fly epoch recreation somehow in a test?

[MichaelHuth]

• add test like TestExportingDataToNWB in HistoricData that executes a Epoch recreation and checks if epochs exists afterwards. This should also cover the fallback paths in the recreation.

[MichaelHuth]

s.baselineFrac and friends to NaN if the following case is hit?

s.baselineFrac is set to NaN when the first LNB read attempt is done unsuccessful in DC_RecreateDataConfigurationResultFromLNB_Indep.

[t-b]

Thanks for the adaptations. LGTM.