OSerror when I tried to use trained model file to predict test data

[weir12]

Hi:
I tried to predict the labels of test set by using the following command for evaluate the performance of custom mod-base model.

   basecall.py --device 0 --modified_base_output basecalls.hdf5 "/root/test_data_7.17/fast5/" "/root/training_data_7.11/training2/model_final.checkpoint" > basecalls.fa

But the following errors occurred:

* Loading model.
* Initializing reads file search.
Traceback (most recent call last):
  File "/root/anaconda3/bin/basecall.py", line 4, in <module>
    __import__('pkg_resources').run_script('taiyaki==4.1.0', 'basecall.py')
  File "/root/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 666, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/root/anaconda3/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1446, in run_script
    exec(code, namespace, namespace)
  File "/root/anaconda3/lib/python3.7/site-packages/taiyaki-4.1.0-py3.7-linux-x86_64.egg/EGG-INFO/scripts/basecall.py", line 198, in <module>
    main()
  File "/root/anaconda3/lib/python3.7/site-packages/taiyaki-4.1.0-py3.7-linux-x86_64.egg/EGG-INFO/scripts/basecall.py", line 142, in main
    recursive=args.recursive))
  File "/root/anaconda3/lib/python3.7/site-packages/taiyaki-4.1.0-py3.7-linux-x86_64.egg/taiyaki/fast5utils.py", line 139, in iterate_fast5_reads
    for y in iterate_files_reads_unpaired(filepaths, read_ids, limit, verbose):
  File "/root/anaconda3/lib/python3.7/site-packages/taiyaki-4.1.0-py3.7-linux-x86_64.egg/taiyaki/fast5utils.py", line 52, in iterate_files_reads_unpaired
    with ont_fast5_api.fast5_interface.get_fast5_file(filepath, 'r') as f5file:
  File "/root/anaconda3/lib/python3.7/site-packages/ont_fast5_api/fast5_interface.py", line 6, in get_fast5_file
    if is_multi_read(filepath):
  File "/root/anaconda3/lib/python3.7/site-packages/ont_fast5_api/fast5_interface.py", line 13, in is_multi_read
    with MultiFast5File(filepath, mode='r') as fast5:
  File "/root/anaconda3/lib/python3.7/site-packages/ont_fast5_api/multi_fast5.py", line 14, in __init__
    self.handle = h5py.File(self.filename, self.mode)
  File "/root/anaconda3/lib/python3.7/site-packages/h5py/_hl/files.py", line 394, in __init__
    swmr=swmr)
  File "/root/anaconda3/lib/python3.7/site-packages/h5py/_hl/files.py", line 170, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 85, in h5py.h5f.open
OSError: Unable to open file (truncated file: eof = 32768, sblock->base_addr = 0, stored_eof = 72986)

Could you help me to resolve this issue?

[tmassingham-ont]

Hello. It could be you are just missing the --recursive option to search below the directory given for fast5 files, or possibly that the files are damaged somehow. Are you able to view the fast5 files using other tools, in particular h5ls from the hdf5-tools package (https://www.hdfgroup.org) or python's h5py package?

Either way, our error message is atrocious and need improving. Sorry.

[weir12]

Thanks,the missing of --recursive is the reason of trouble.

[weir12]

Hi:
I had a similar problem again.

self.handle = h5py.File(self.filename, self.mode)
  File "/home/anaconda3/lib/python3.7/site-packages/h5py/_hl/files.py", line 394, in __init__
    swmr=swmr)
  File "/home/anaconda3/lib/python3.7/site-packages/h5py/_hl/files.py", line 170, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 85, in h5py.h5f.open
OSError: Unable to open file (truncated file: eof = 68608, sblock->base_addr = 0, stored_eof = 494195)

I have added "--recursive".So I guess some fast5 files are corrupted in input folder .For another fast5 folder, no error is reported.

[myrtlecat]

Yes, I think this indicates that some fast5s are either corrupted, or may be in use by another process. Is there anything else running that might be writing to these files while you are trying to basecall them?

It's not very useful that the error message doesn't indicate which file caused the failure. We will try to improve the error reporting in the next patch.

[weir12]

Thanks for your help. @myrtlecat
I used lsof and fuser to make sure that the folder was not being occupied by another process.
On the other hand,The OSError didn't appear until about half an hour later after the software starts running.So my guess is that a corrupted fast5 file was found and then process crashed.
Is there a way to skip/remove corrupted fast5 files?