add input type fastq.gz and fq.gz for Illumina and HiC reads

[reichan1998]

To solve issue #88

PR checklist

• This comment contains a description of changes (with reason).
• If you've fixed a bug or added code that should be tested, add tests!
• If you've added a new tool - have you followed the pipeline conventions in the contribution docs
• Make sure your code lints (nf-core lint).
• Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
• Usage Documentation in docs/usage.md is updated.
• Output Documentation in docs/output.md is updated.
• CHANGELOG.md is updated.
• README.md is updated (including new tool citations and authors/contributors).

[tkchafin]

For this one since align_illumina_fastq is just copying all of the steps (except for CRAM -> fastq conversion), it would be simpler to using a branch operation to determine if the SAMTOOLS_FASTQ step is needed (true for CRAM but false if input type is FASTQ).

For example you can see further on in align_short.nf where there is a branch operation to determine whether SAMTOOLS_MERGE will be run:

    // Collect all BWAMEM2 output by sample name
    BWAMEM2_MEM.out.bam
    | map { meta, bam -> [['id': meta.id.split('_')[0..-2].join('_'), 'datatype': meta.datatype], meta.read_count, bam] }
    | groupTuple( by: [0] )
    | map { meta, read_counts, bams -> [meta + [read_count: read_counts.sum()], bams] }
    | branch {
        meta, bams ->
            single_bam: bams.size() == 1
            multi_bams: true
    }
    | set { ch_bams }


    // Merge, but only if there is more than 1 file
    SAMTOOLS_MERGE ( ch_bams.multi_bams, [ [], [] ], [ [], [] ] )
    ch_versions = ch_versions.mix ( SAMTOOLS_MERGE.out.versions.first() )

The branch section of this is creating two branches: single_bam if the sample has a single BAM file, or multi_bams, which catches all cases where there is not a single BAM file:

    | branch {
        meta, bams ->
            single_bam: bams.size() == 1
            multi_bams: true
    }

The next step is we run SAMTOOLS_MERGE only on the multi_bams branch, and then combine the outputs with the single_bam branch. The process that you will need to implement for FASTQ vs CRAM inputs will be similar

[tkchafin]

Also note that ALIGN_SHORT is actually used for both Illumina and HiC reads, so any changes in documentation for Illumina also applies to HiC:

include { ALIGN_SHORT as ALIGN_HIC      } from '../subworkflows/local/align_short'
include { ALIGN_SHORT as ALIGN_ILLUMINA } from '../subworkflows/local/align_short'

[reichan1998]

Thank you very much for your instructions, @tkchafin. I noticed that ALIGN_SHORT is used for both Illumina and HiC reads. Therefore, I decided to add a condition to branch reads with the fastq.gz format in readmapping.nf, so it will leave the ALIGN_HIC unchanged. Your approach is certainly more efficient and structured, and I will adjust as you described.

[tkchafin]

I think we can allow the FASTQ option for both HiC and Illumina, so it might be best to keep the branching inside of align_short (the feature request actually was about HiC #88 )

[reichan1998]

I'm so sorry for the oversight and misunderstanding regarding the FASTQ option for HiC. I've made the changes as per your instructions . Thank you for your review and guidance.

[reichan1998]

Hi @tkchafin, I've made the changes following your instructions. Could you please help me review them? I really appreciate your guidance. Thank you!

[tkchafin]

Hi @reichan1998 thanks for these changes. I've been off sick the past few days, but will have a look soon.

[tkchafin]

Great job on these changes @reichan1998! The next step is we should incorporate the new input type into the test profile. If you look in conf/test.config you can see the settings and sample sheet used when you run with -profile test. This points you to assets/samplesheet_s3.csv which gives paths to files saved remotely in AWS S3 buckets.

Since there are already two, would suggest we try converting one of the Illumina samples from CRAM to FASTQ so tests will cover both branches of your new align_short.nf:

mMelMel2,illumina,https://tolit.cog.sanger.ac.uk/test-data/Meles_meles/genomic_data/mMelMel2/illumina/31231_4%231.subset.cram,

You can download the file with curl -L -O mMelMel2/illumina/31231_4%231.subset.cram and then convert to fastq using samtools fastq.

First try manually editing the sample sheet to point to your new local file to test, and if this passes you can place the file onto the farm and give me the path and I'll add it to the remotely hosted test files.

[tkchafin]

I have added the file to the hosted test_data, the path is https://tolit.cog.sanger.ac.uk/test-data/Meles_meles/genomic_data/mMelMel2/illumina/31231_4%231.subset.fastq so you can put this into the assets/samplesheet_s3.csv for mMelMel2,illumina to test the pipeline with the fastq input option for that sample