Bad hk data in range data prevents loading entire range of data.

[jeffiuliano]

Bad frames? prevent loading of any range of data with load_range that includes that bad data. A more graceful failure that just skips the bad part would be preferable.

~/so3g_install/lib/python3.7/site-packages/so3g/hk/getdata.py in load_range(start, stop, fields, alias, data_dir, config, pre_proc_dir, pre_proc_mode, strict)
    782             if t >= start_ctime-3600 and t <=stop_ctime+3600:
    783                 hk_logger.debug('Processing {}'.format(base+'/'+file))
--> 784                 hksc.process_file_with_cache( base+'/'+file)
    785 
    786 

~/so3g_install/lib/python3.7/site-packages/so3g/hk/getdata.py in process_file_with_cache(self, filename)
    612         else:
    613             hksc = HKArchiveScanner()
--> 614             hksc.process_file(filename)
    615             # Make dirs if needed
    616             if not os.path.exists( os.path.dirname(path) ):

~/so3g_install/lib/python3.7/site-packages/so3g/hk/getdata.py in process_file(self, filename, flush_after)
    577             info = {'filename': filename,
    578                     'byte_offset': reader.Tell()}
--> 579             frames = reader.Process(None)
    580             assert len(frames) <= 1
    581             if len(frames) == 0:

RuntimeError: Failed to read 4 bytes from input stream! Read 2

[jeffiuliano]

More info on where the bad frames are: After wrapping line 579 above in a try/except so it doesn't just break things (and printing the scanner counter), I get a useful error message:

ERROR (G3IndexedReader): Exception raised while reading file /mnt/so1/data/penn-latr/hk/16324/1632450421.g3 (G3IndexedReader.cxx:85 in virtual void G3IndexedReader::Process(G3FramePtr, std::deque<boost::shared_ptr<G3Frame> >&))
reader error, counter: 936
ERROR (G3IndexedReader): Exception raised while reading file /mnt/so1/data/penn-latr/hk/16460/1646040049.g3 (G3IndexedReader.cxx:85 in virtual void G3IndexedReader::Process(G3FramePtr, std::deque<boost::shared_ptr<G3Frame> >&))
reader error, counter: 603