tfq support for distmap upload

[robmaz]

Sooner or later I would also like to move from Ram's format where uninformative fastq lines are removed before putting the records on one line to my more straight-forward "tabbed fastq" format where nothing is removed and records are jammed together like this (for paired-end):

samtools fastq $b | paste - - - - - - - - | hadoop fs -put - ${b%.bam}.tfq

(or ... | mc4 | hadoop fs ... ....tfq.mc4 with mc4 compression.) Which is much less hassle to handle through command line tools.

While readtools support for this is not so urgent, maybe it would not be so difficult to already include an option to produce this format?

[magicDGS]

@robmaz - doing that sounds like you will keep a lot of not needed information while uploading (such a duplicated read names, and quality headers that contains nothing).

Are you sure that this tabbed format is desirable? It might increase the size of the files, and ReadTools have already implemented a parser for the current distmap format. I think that the proper way to go is to directly have BAM files in HDFS: the input format can be handled by Hadoop-BAM and there are even splitting indexes for BAM files to fast split of records in the same partition...

[robmaz]

I think with any compressor this unneeded lines will basically disappear anyway, and I think it is desirable to easily generate and read the input stored in hdfs with alternative tools.

While I agree that going for BAMs might be preferable in the long run, let me highlight the two major current issues:

• Hadoop-BAM has no release for Hadoop 2.7.x, only 2.2. It requires building, probably messy like everything Hadoop-related, and may not even work.
• Since bwa mem does not support BAM input, we'd need a custom InputFormat class, which we currently don't have. This also needs to take care of the header issue (I think you said that headers may be required in the near future? It is curious that the BAM-heavy GATK people who also use bwa mem as their default mapper have not resolved this issue, so maybe something is coming from there?)

The tabbed fastq thing in contrast works already, and to get mc4 support in I just need to restart the cluster at some point.

[robmaz]

In any case, this is a minor feature and certainly not urgent.

[magicDGS]

I see your point, and it might be good to have the tabbed input sooner than later if you want to move to an easier to manage format. Regarding the issues with BAM:

• The GATK team uses Hadoop-BAM and they are using Hadoop >= 2.7, and they are just pulling the library from maven central. So probably there is no issue for using it with the current version. In addition, there is now a discussion in which I am involved about the governance of Hadoop-BAM and how to proceed with its development. Thus, it looks promising for the developmental of Distmap.
• Although bwa mem does not support BAM input, the InputFormat class can still be the SAM/BAM abstraction from Haddop-BAM. As it is done now, with the custom distmap format, the map task in the MapReduce pipeline is converting the input into a format understandable by the mapper. Thus, the headerless-bam part passed from Hadoop-BAM can be converted to FASTQ (using for example Picard or samtools) and then passed to the mapper; afterwards, Picard can merge the unaligned input with the aligned output, and thus keep all the information.

The GATK team does not care about the bwa mem implementation; in their pre-GATK4 pipeline they were converting to FASTQ before mapping and then merging with Picard; after GATK4 they have a Spark version from unmapped BAM to mapped reads with a JNI for bwa.

In addition, GATK is using the Hadoop-BAM library for almost every HDFS/Spark related piece of code, so that means that it can work properly here too.

The real way of moving forward the Distmap to accept any kind of input and do it in a consistent way is to implement the framework in ReadTools. Or maybe better, implement a Sparkified version of Distmap in ReadTools. Anyway, that it is much more difficult than just changing the input format If you really think that it will be useful, I can spend some of my free-developmental time to implement a hidden and advance argument for output this new format (--new-format).

[magicDGS]

@robmaz - one question about this: what will happen if the FASTQ that is parsed by paste contains already tabs? This will fail!

I am looking into this and also into the 4mc codec...

[robmaz]

I don't think the sequence or the quality string can contain tabs, but maybe
the descriptions in Lines 1 and 3? Fortunately none of my fastqs had them.
Anyway, the obvious solution would be to mask them before pasting and
unmasking them after breaking, and I guess the current format must solve
the same problem.

[magicDGS]

Usually, in the first line is not recommended to have them (and ReadTools is passing it to a comment tag), but in the quality header I found a lot of weird stuff in the past. The current format does not need to mask any of that.

Regarding the masking, do you mean to percent-encode the tabs in the quality header? That will be a proper way of handling it, but then it requires post-processing anyway (and thus, the major improvement of easy-to-parse does not hold...)

[magicDGS]

@robmaz - do you think that it is really important to keep the quality header in this case? What's about using @{readName}\t{sequence}\t+\t{quality} for single end and @{readName1}\t{sequence1}\t+\t{quality1\t@{readName2}\t{sequence2}\t+\t{quality2}}, ignoring the quality header?

This is still having the same advantages of using paste, but without including unnedded information in the quality header. With ReadTools, it is ensure that the read name will contain the information required for processing (Illumina-encoded name, using # to separate barcode and / to separate pair-end information).

If this is good enough for you, I can re-factor the code in the distmap output to detect the .tfq extension and write down that format; otherwise, it will output the current one. Another option is a extension-independent format and an advance parameter. Let me know what is better for your case...

[robmaz]

Not at all, since this is only used to construct input files for the mappers. But what I now did alternatively after you warned me of the tabs was to change all tabs to spaces before paste-ing it, i.e., ... | tr '\t' ' ' | paste - - - - - - - - ... It is not very difficult to produce the format you suggest in a shell one-liner, but I think this is even simpler. Cheers Rupert 2018-02-18 13:13 GMT+01:00 Daniel Gómez-Sánchez <notifications@github.com>:

…

@robmaz <https://github.com/robmaz> - do you think that it is really important to keep the quality header in this case? What's about using @{readName}\t{sequence}\t+\t{quality} for single end and @{readName1}\t{sequence1}\t+\t{quality1\t@{readName2}\t{ sequence2}\t+\t{quality2}}, ignoring the quality header? This is still having the same advantages of using paste, but without including unnedded information in the quality header. With ReadTools, it is ensure that the read name will contain the information required for processing (Illumina-encoded name, using # to separate barcode and / to separate pair-end information). If this is good enough for you, I can re-factor the code in the distmap output to detect the .tfq extension and write down that format; otherwise, it will output the current one. Another option is a extension-independent format and an advance parameter. Let me know what is better for your case... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#404 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Ad_FfJFxME4DyPk-i8qyEqIO6TGdQQCmks5tWBPigaJpZM4SFGha> .