2025 corrupt evio file causes segfault with version set 7.2.0

[nsjarvis]

Run 133245 file 0 gives me this:

13:47:00.746 [error] Slot from SSP/DIRC event header does not match slot from last block header (17 != 3)
13:47:00.996 [error] Bad SSP Data! DEVIOWorkerThread::ParseSSPBank()
13:47:01.007 [fatal] Segfault detected!

and another run over the same data in my batch file which hadn't detected the crash gave this slightly different version

13:48:27.600 [error] Slot from SSP/DIRC event header does not match slot from last block header (17 != 3)
13:48:27.600 [error] Bad SSP Data! DEVIOWorkerThread::ParseSSPBank()
ERROR: INCORRECT INPUT OBJECT TO DTTabUtilities::Convert_DigiTimeToNs_F1TDC(). RETURNING NaN.
13:48:27.609 [fatal] Segfault detected!
13:48:27.624 [fatal] Segfault detected!
13:48:27.624 [fatal] Segfault detected!

[sdobbs]

To skip these errors, you can use the flag -PEVIO:SKIP_SSP_FORMAT_ERROR=1

[nsjarvis]

Thanks! Why isn't that on by default?

[sdobbs]

The current policy is to throw fatal errors by default so that data formatting errors like this are readily apparent, and not hidden in large log files, for example.

[nsjarvis]

It would be better to fix the evio parser to skip the event.

[zihlmann]

Isn't this error caused by corrupted data most likely caused by a hardware issue? In that case you want to know that because rebooting that hardware will most likely cure the problem. If not this problem will persist. Other halls with high radiation regularly reboot all their equipment to forego such effect. We encounters a similar issue last run period where the reconstruction failed and the shift crew run blind not realizing that the actual cause was corrupted data from hardware. We may want to increase the verbosity of the error message and in addition provide better instructions for the shift crew.
For actual reconstruction and rest production one may want to consider other options.

[nsjarvis]

I don't know the cause but that's an excellent suggestion, if we put some effort into propagating the errors up to somewhere where the shift crew will see them easily. We could be much more consistent about this in the decoder, as that deals with other hardware errors more quietly. The crash-errors are not so obvious in the monitoring launches as those use 5 files.
Igal reports a usual failure rate of 0.2% from the REST production, it would be good to have a group of us track down the cause of that.

[nsjarvis]

Run 132925 file 000 gives a new (to me) error:

HDEVIO::MapEvents warning - Attempt to swap bank with len<2 (len=0 header=0 pos=7250551688 tellg=7250551716 i=62)
HDEVIO::MapEvents warning - Attempt to swap bank with len<2 (len=0 header=0 pos=7250551692 tellg=7250551720 i=63)
HDEVIO::MapEvents warning - Attempt to swap bank with len<2 (len=0 header=0 pos=7250551696 tellg=7250551724 i=64)
...
HDEVIO::MapEvents warning - Attempt to swap bank with len<2 (len=0 header=0 pos=7250551920 tellg=7250551948 i=120)
HDEVIO::MapEvents warning - Attempt to swap bank with len<2 (len=0 header=0 pos=7250551924 tellg=7250551952 i=121)
01:32:51.453 [error] EVIO bank length word is zero in swap_bank!
01:32:51.459 [fatal] Segfault detected!

Update: with version set 7.4.0 it gives the error message and exits properly, no segfault.

      FADC250 Block Header: slot=20 (0x85077328)
      FADC250 Event Header: itrigger=1731025, rocid=12, slot=20) (0x951a69d1)
      FADC250 Pulse Data (0xc831018b) channel=2 pedestal=395 event within block=6
      FADC250 Pulse Data word 2(0x406ed006)  integral=1773
      FADC250 Pulse Data word 3(0x9a78450)  course_time=77 fine_time=15 pulse_peak=138
      FADC250 Pulse Data (0xc85d018e) channel=10 pedestal=398 event within block=11
      FADC250 Pulse Data word 2(0x4071200b)  integral=1810
      FADC250 Pulse Data word 3(0x73003d8)  course_time=57 fine_time=32 pulse_peak=123
      FADC250 Pulse Data (0xc874818d) channel=9 pedestal=397 event within block=14
      FADC250 Pulse Data word 2(0x40696004)  integral=1686
      FADC250 Pulse Data word 3(0x2da8380)  course_time=22 fine_time=53 pulse_peak=112
      FADC250 Pulse Data (0xc876818f) channel=13 pedestal=399 event within block=14
      FADC250 Pulse Data word 2(0x406e3006)  integral=1763
      FADC250 Pulse Data word 3(0x2d00440)  course_time=22 fine_time=32 pulse_peak=136
      FADC250 Pulse Data (0xc93a8194) channel=5 pedestal=404 event within block=39
      FADC250 Pulse Data word 2(0x4073e00c)  integral=1854
      FADC250 Pulse Data word 3(0x30b0458)  course_time=24 fine_time=22 pulse_peak=139
      FADC250 Pulse Data (0xc93b018f) channel=6 pedestal=399 event within block=39
      FADC250 Pulse Data word 2(0x411d700f)  integral=4567
      FADC250 Pulse Data word 3(0x30713b8)  course_time=24 fine_time=14 pulse_peak=631
      FADC250 Pulse Data word 2(0x4037a807)  integral=890
      FADC250 Pulse Data word 3(0xb950398)  course_time=92 fine_time=42 pulse_peak=115
      FADC250 Pulse Data (0xc93b8194) channel=7 pedestal=404 event within block=39
      FADC250 Pulse Data word 2(0x406ad006)  integral=1709
      FADC250 Pulse Data word 3(0x2f983a0)  course_time=23 fine_time=51 pulse_peak=116
      FADC250 Block Trailer (0x8d00001a)  iptr=0x0x7f8adc08cb90

...

21:50:28.367  [info] ============================== Event 1059320 ==============================
21:50:28.367  [info] 
21:50:28.397 [error] EVIO bank length word is zero in swap_bank!

Maybe there is more than one problem in the evio file.

generates many of these errors

HDEVIO::MapEvents warning - Attempt to swap bank with len<2 (len=0 header=0 pos=7250551684 tellg=7250551712 i=61)

but it exits properly. However if I ask for 100 events instead of 42 then it segfaults.

11:46:31.690  [info] ============================== Event 1059135 ==============================
11:46:31.690  [info] 
11:46:31.690 [fatal] Segfault detected!
11:46:31.691 [fatal] Hard exit due to segmentation fault! Backtrace:
11:46:31.691 [fatal] 
11:46:31.691 [fatal]    0: void JObject::GetAssociatedAncestors<Df250PulseData>(std::set<JObject const*, std::less<JObject const*>, std::allocator<JObject const*> >&, int&, std::set<Df250PulseData const*, std::less<Df250PulseData const*>, std::allocator<Df250PulseData const*> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) const
11:46:31.691 [fatal]       hd_root:b29bb7
11:46:31.691 [fatal] 
11:46:31.691 [fatal]    1: void JObject::GetAssociatedAncestors<Df250PulseData>(std::set<JObject const*, std::less<JObject const*>, std::allocator<JObject const*> >&, int&, std::set<Df250PulseData const*, std::less<Df250PulseData const*>, std::allocator<Df250PulseData const*> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) const
11:46:31.691 [fatal]       hd_root:b29c78
11:46:31.691 [fatal] 
11:46:31.691 [fatal]    2: void JObject::Get<Df250PulseData>(std::vector<Df250PulseData const*, std::allocator<Df250PulseData const*> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int) const
11:46:31.691 [fatal]       hd_root:b29ec4
11:46:31.691 [fatal] 
11:46:31.691 [fatal]    3: JEventSource_EVIOpp::EmulateDf250Firmware(DParsedEvent*)
11:46:31.691 [fatal]       hd_root:f41f67
11:46:31.691 [fatal] 
11:46:31.691 [fatal]    4: JEventSource_EVIOpp::GetObjects(std::shared_ptr<JEvent const> const&, JFactory*)
11:46:31.691 [fatal]       hd_root:f43c3b
11:46:31.691 [fatal] 
11:46:31.691 [fatal]    5: ???
11:46:31.691 [fatal]       /home/gluex/gluex_topj2/jana/jana_2.4.3^root6.32.20/Linux_Alma9-x86_64-gcc11.5.0/lib/libJANA.so:1f4f65
11:46:31.691 [fatal] 
11:46:31.691 [fatal]    6: ???
11:46:31.691 [fatal]       /home/gluex/gluex_topj2/jana/jana_2.4.3^root6.32.20/Linux_Alma9-x86_64-gcc11.5.0/lib/libJANA.so:1f6117
11:46:31.691 [fatal] 
11:46:31.691 [fatal]    7: JFactory::Create(JEvent const&)
11:46:31.691 [fatal]       /home/gluex/gluex_topj2/jana/jana_2.4.3^root6.32.20/Linux_Alma9-x86_64-gcc11.5.0/lib/libJANA.so:1f548f
11:46:31.691 [fatal] 
11:46:31.691 [fatal]    8: JFactoryT<DBadHit>* JEvent::Get<DBadHit>(std::vector<DBadHit const*, std::allocator<DBadHit const*> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) const
11:46:31.691 [fatal]       /home/njarvis/git/halld_recon/Linux_Alma9-x86_64-gcc11.5.0/plugins/bad_hits.so:1533a
11:46:31.691 [fatal] 
11:46:31.691 [fatal]    9: JEventProcessor_bad_hits::Process(std::shared_ptr<JEvent const> const&)
11:46:31.691 [fatal]       /home/njarvis/git/halld_recon/Linux_Alma9-x86_64-gcc11.5.0/plugins/bad_hits.so:99f3
11:46:31.691 [fatal] 
11:46:31.691 [fatal]   10: void jana::components::JComponent::CallWithJExceptionWrapper<JEventProcessor::DoLegacyProcess(std::shared_ptr<JEvent const> const&)::{lambda()#3}>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, JEventProcessor::DoLegacyProcess(std::shared_ptr<JEvent const> const&)::{lambda()#3})
11:46:31.691 [fatal]       hd_root:45fdcf
11:46:31.691 [fatal] 
11:46:31.691 [fatal]   11: JEventProcessor::DoLegacyProcess(std::shared_ptr<JEvent const> const&)
11:46:31.691 [fatal]       hd_root:460295
11:46:31.691 [fatal] 
11:46:31.691 [fatal]   12: JEventMapArrow::fire(JEvent*, std::array<std::pair<JEvent*, int>, 2ul>&, unsigned long&, JArrow::FireResult&)
11:46:31.691 [fatal]       /home/gluex/gluex_topj2/jana/jana_2.4.3^root6.32.20/Linux_Alma9-x86_64-gcc11.5.0/lib/libJANA.so:217282
11:46:31.691 [fatal] 
11:46:31.691 [fatal]   13: JExecutionEngine::RunWorker(JExecutionEngine::Worker)
11:46:31.691 [fatal]       /home/gluex/gluex_topj2/jana/jana_2.4.3^root6.32.20/Linux_Alma9-x86_64-gcc11.5.0/lib/libJANA.so:20b43d
11:46:31.691 [fatal] 
11:46:31.691 [fatal]   14: void std::__invoke_impl<void, void (JExecutionEngine::*)(JExecutionEngine::Worker), JExecutionEngine*, JExecutionEngine::Worker>(std::__invoke_memfun_deref, void (JExecutionEngine::*&&)(JExecutionEngine::Worker), JExecutionEngine*&&, JExecutionEngine::Worker&&)
11:46:31.691 [fatal]       /home/gluex/gluex_topj2/jana/jana_2.4.3^root6.32.20/Linux_Alma9-x86_64-gcc11.5.0/lib/libJANA.so:212f8d
11:46:31.691 [fatal] 
11:46:31.691 [fatal]   15: std::__invoke_result<void (JExecutionEngine::*)(JExecutionEngine::Worker), JExecutionEngine*, JExecutionEngine::Worker>::type std::__invoke<void (JExecutionEngine::*)(JExecutionEngine::Worker), JExecutionEngine*, JExecutionEngine::Worker>(void (JExecutionEngine::*&&)(JExecutionEngine::Worker), JExecutionEngine*&&, JExecutionEngine::Worker&&)
11:46:31.691 [fatal]       /home/gluex/gluex_topj2/jana/jana_2.4.3^root6.32.20/Linux_Alma9-x86_64-gcc11.5.0/lib/libJANA.so:212eae
11:46:31.691 [fatal] 
11:46:31.691 [fatal]   16: void std::thread::_Invoker<std::tuple<void (JExecutionEngine::*)(JExecutionEngine::Worker), JExecutionEngine*, JExecutionEngine::Worker> >::_M_invoke<0ul, 1ul, 2ul>(std::_Index_tuple<0ul, 1ul, 2ul>)
11:46:31.691 [fatal]       /home/gluex/gluex_topj2/jana/jana_2.4.3^root6.32.20/Linux_Alma9-x86_64-gcc11.5.0/lib/libJANA.so:212de1
11:46:31.691 [fatal] 
11:46:31.691 [fatal]   17: std::thread::_Invoker<std::tuple<void (JExecutionEngine::*)(JExecutionEngine::Worker), JExecutionEngine*, JExecutionEngine::Worker> >::operator()()
11:46:31.691 [fatal]       /home/gluex/gluex_topj2/jana/jana_2.4.3^root6.32.20/Linux_Alma9-x86_64-gcc11.5.0/lib/libJANA.so:212d7e
11:46:31.691 [fatal] 
11:46:31.691 [fatal]   18: std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (JExecutionEngine::*)(JExecutionEngine::Worker), JExecutionEngine*, JExecutionEngine::Worker> > >::_M_run()
11:46:31.691 [fatal]       /home/gluex/gluex_topj2/jana/jana_2.4.3^root6.32.20/Linux_Alma9-x86_64-gcc11.5.0/lib/libJANA.so:212d62
11:46:31.691 [fatal] 
11:46:31.691 [fatal]   19: ???
11:46:31.691 [fatal]       /lib64/libstdc++.so.6:dbae4
11:46:31.691 [fatal] 
11:46:31.691 [fatal]   20: ???
11:46:31.691 [fatal]       /lib64/libc.so.6:8b2ea
11:46:31.691 [fatal] 
11:46:31.691 [fatal]   21: ???
11:46:31.691 [fatal]       /lib64/libc.so.6:1103c0
11:46:31.691 [fatal] 

Events processed started at 1058961 which is a few blocks before the problem event.

[nsjarvis]

132066 file 000

14:00:00.949 [error] WARNING: Bank length word exceeds valid words in buffer (3427832433 > 3426783839)
14:00:00.987 [fatal] Segfault detected!
14:00:00.997 [fatal] Hard exit due to segmentation fault! Backtrace:
14:00:00.997 [fatal] 
14:00:00.997 [fatal]    0: __dynamic_cast
14:00:00.997 [fatal]       /lib64/libstdc++.so.6:abfcf
14:00:00.997 [fatal] 
14:00:00.997 [fatal]    1: std::vector<DF1TDCHit const*, std::allocator<DF1TDCHit const*> > JObject::Get<DF1TDCHit>() const
14:00:00.997 [fatal]       hd_root:f7624e

Update: With the new version set 7.4.0 I see this

      FADC250 Block Header: slot=20 (0x8505f528)
      FADC250 Event Header: itrigger=294433, rocid=12, slot=20) (0x95047e21)
      FADC250 Pulse Data (0xc851018c) channel=2 pedestal=396 event within block=10
      FADC250 Pulse Data word 2(0x4085200e)  integral=2130
      FADC250 Pulse Data word 3(0x28a04b0)  course_time=20 fine_time=20 pulse_peak=150
      FADC250 Pulse Data (0xc869018f) channel=2 pedestal=399 event within block=13
      FADC250 Pulse Data word 2(0x4081b00d)  integral=2075
      FADC250 Pulse Data word 3(0x1a803a0)  course_time=13 fine_time=16 pulse_peak=116
      FADC250 Event Trailer, Data not Valid, or Filler word (15) (0xfd0000ed)
      FADC250 Block Trailer (0x8d00000a)  iptr=0x0x7fc8b0dac6d8
11:16:37.677  [info] 
11:16:37.677  [info] ============================== Event 757281 ==============================

The "[info]===Event" lines are generated by a plugin, the output continues with only these until

11:16:37.704  [info] ============================== Event 757320 ==============================
11:16:37.704  [info] 
11:16:37.876  [info] Status: 378683 events processed at 80.0 Hz (338.5 Hz avg)
11:16:38.376  [info] Status: 378683 events processed at 0.0 Hz (338.3 Hz avg)
11:16:38.842 [error] WARNING: unknown bank type (0xff)

and there it exits. The warning message comes from https://github.com/JeffersonLab/halld_recon/blob/c5ae4c87765454cd3ea6aa1e6f9768431d46ddce/src/libraries/DAQ/swap_bank.cc#L89
Events 757241 to 757280 do not reach the plugin.

[sdobbs]

Agreed that it would be good to go through and figure out what to do about each of these errors.
At least as a start, they should all throw exceptions and not crash hard.

[nsjarvis]

I made this branch https://github.com/JeffersonLab/halld_recon/tree/nsj_badhit