Not sure this is exiting loop

[hyphaltip]

funannotate/funannotate/library.py

Line 594 in 17a353a

log.debug('R1 header: {}'.format(read1[0]))

this was the RNAseq

@A00953:17:HTJKFDSXX:4:1101:6696:1000 1:N:0:GTGTCGGA+GCTTGCGC
GTCTGGCCATCCTTGGAGATACCAGCTTCGAACTCACCAGTACCGGCAGCAATGATAAGGATAGCGCAATCAGCCTGGGAGGTACCAGTAATCATGTTCTT
+
FFFFFFFFFFFF:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

and R2

@A00953:17:HTJKFDSXX:4:1101:6696:1000 2:N:0:GTGTCGGA+GCTTGCGC
CTGGAAGTTCGAGACCCCCAAGTACCACGTCACTGTCATCGATGCCCCTGGCCATCGTGACTTCATCAAGAACATGATTACTGGTACCTCCCAGGCTGATT
+
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFFF

Not sure why this isn't triggering the loop break as expected.

[hyphaltip]

But otherwise with the log statement as they are, it printed out every line in the RNASeq file...

[nextgenusfs]

Oh, I think I had that there for debugging and never removed it.....

[nextgenusfs]

Those headers look okay to me -- are there two spaces in header or something like that?

[hyphaltip]

no one space... I will try to cook a simpler example so I can test out - I also processed the reads so they are all /1 and /2 ending but it seemed to have still failed but I'll try a new run just to be sure.

[nextgenusfs]

maybe this makes more sense:

def CheckFASTQandFix(forward, reverse, cpus=2):
    from Bio.SeqIO.QualityIO import FastqGeneralIterator

    # open and check first header, if okay exit, if not fix
    file1 = FastqGeneralIterator(zopen(forward, 'rt'))
    file2 = FastqGeneralIterator(zopen(reverse, 'rt'))
    check = True
    written2log = 0
    for read1, read2 in zip(file1, file2):
        # see if index is valid
        if ' ' in read1[0] and ' ' in read2[0]:
            # std illumina, exit
            if read1[0].split(' ', 1)[1].startswith('1') and read2[0].split(' ', 1)[1].startswith('2'):
                break
        else:
            if read1[0].endswith('/1') and read2[0].endswith('/2'):  # also acceptable
                break
            else:  # it is not okay missing paired information
                check = False
                if written2log < 4:
                    log.debug('R1 header: {}'.format(read1[0]))
                    log.debug('R2 header: {}'.format(read2[0]))
                written2log += 1
                break
    file1.close()
    file2.close()

But I thought those breaks should just exit -- ie not sure it is necessary to read the whole file?

[hyphaltip]

its weird because this is not evaluating to true when it should - this logic should be catching and if it isn't then we also need to check why not. It's actually not getting to the last else statement

if read1[0].split(' ')[1].startswith('1') and read2[0].split(' ')[1].startswith('2'):

I added some logging

[09/28/20 20:13:31]: read1[0].split(' ')[1]=1:N:0:GTGTCGGA+GCTTGCGC
[09/28/20 20:13:31]: read2[0].split(' ')[1]=1:N:0:GTGTCGGA+GCTTGCGC

still debugging...

[hyphaltip]

oh shoot I see second read has 1:N not 2:N - let me make sure that's not a typo in making the test file.

[hyphaltip]

Not that's the problem read1 and read2 are actually the same read... not in the data file - the

[nextgenusfs]

Its working then! ;) not properly paired!

[hyphaltip]

no the files are actually right...

[hyphaltip]

hahh okay I think I know... ! my script:

--left $RNAFOLDER/${RNASEQSET}_R1.fq.gz --right $RNAFOLDER/${RNASEQSET}_R1.fq.gz

instead of

--left $RNAFOLDER/${RNASEQSET}_R1.fq.gz --right $RNAFOLDER/${RNASEQSET}_R2.fq.gz

[hyphaltip]

so yeah the code is working but we should have it stop after a few rounds of testing in that loop I think... Or do you want it to look an entire file?

            # std illumina, exit
            if read1[0].split(' ')[1].startswith('1') and read2[0].split(' ')[1].startswith('2'):
                break
            else:   # add this?
		check =	False # add this?
                break # add this

[nextgenusfs]

It wasn't working quite properly anyway though? I'm not sure it has to check the whole thing -- but never know what kind of data people with throw at it, ie concatenated elsewhere, half good, half bad?

[hyphaltip]

yeah it was breaking at the next steps anyways in trinity but figuring out what was the problem...

I just think we probably shouldn't have this loop run through the entire file -- but basically this would be a case where the pairing wasn't respected, I think it is okay to say "this is borked, check you file the 1:XX and 2:XX as expected in this illumina format are not coded correctly" -- don't expect funannotate to fix it just to flag the problem and quit.