OSError: [Errno 116] Stale file handle

[johnnyasena]

Hi,

I attempted to run Taiyaki using CPU but received the following error. Please, can you give me advice on how to fix this issue?

Traceback (most recent call last):
File "/home/s953272/sw/taiyaki/bin/train_flipflop.py", line 530, in
main()
File "/home/s953272/sw/taiyaki/bin/train_flipflop.py", line 456, in main
None if args.gradient_cap_fraction is None else gradient_cap)
File "/home/s953272/TAIYAKI_ENV/lib/python3.6/site-packages/taiyaki-5.0.1-py3.6-linux-x86_64.egg/taiyaki/helpers.py", line 233, in record
self.write("{:5.4f}\t{:5.4f}\t".format(loss, gradientnorm))
File "/home/s953272/TAIYAKI_ENV/lib/python3.6/site-packages/taiyaki-5.0.1-py3.6-linux-x86_64.egg/taiyaki/helpers.py", line 227, in write
self.fh.write(s.encode('utf-8'))
OSError: [Errno 116] Stale file handle
Command exited with non-zero status 1

Here is the command I used.

/home/s953272/sw/taiyaki/bin/train_flipflop.py --overwrite --size 256 --stride 2 --winlen 19 --device cpu --outdir /scratch-large/5-biannual/D8661/nanopore_basecalling/guppy/taiyaki/30Jan17_C0576_20kb /home/s953272/taiyaki/models/mGru_flipflop.py /scratch-large/5-biannual/D8661/nanopore_basecalling/guppy/minimap2/30Jan17_C0576_20kb/mapped_reads.hdf5"

[tmassingham-ont]

The error is occurred when writing to one of the log files, which is kept open throughout the training process. The only time I've encountered this kind of error is when the file being written to has been deleted, or otherwise invalidated after being opened. Generally a filesystem issue.

How far through training did the error happen, and are you able to reproduce?

Thanks,

TimM.

[johnnyasena]

Hi Tim, I am rerunning Taiyaki using GPU to see if this is reproducible. I was running with CPU. I will update you. Thank you for your swift response! Sincerely, Johnny Sena, Ph.D. Bioinformatics Research Associate and Lab Coordinator National Center for Genome Resources 2935 Rodeo Park Drive E Santa Fe, NM 87505 Office: (505) 995 4437 Fax: (505) 995 4461 Email: jsena@ncgr.org

…

On Tue, Jan 7, 2020 at 1:32 PM Tim Massingham ***@***.***> wrote: The error is occurred when writing to one of the log files, which is kept open throughout the training process. The only time I've encountered this kind of error is when the file being written to has been deleted, or otherwise invalidated after being opened. Generally a filesystem issue. How far through training did the error happen, and are you able to reproduce? Thanks, TimM. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#66?email_source=notifications&email_token=ADA5HL5I23JVHHWBAFSJLO3Q4TRE7A5CNFSM4KD3BFT2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIKFQRA#issuecomment-571758660>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADA5HLZDJUZJ3SOORCPWGS3Q4TRE7ANCNFSM4KD3BFTQ> .