CWLtool issue with passing a list of scatter/gathered outputs into a program that accepts a list

[jstjohn]

This works:

#!/usr/bin/env cwl-runner
class: Workflow
requirements:
  - class: ScatterFeatureRequirement
inputs:
  - id: "#mutation"
    type: string
  - id: "#normalin"
    type: File
  - id: "#tumorin"
    type: File
outputs:
  - id: "#outfile"
    type: File
    source: "#wc.outfile"
steps:
  - id: "#greptumor"
    run: {import: grep.cwl.yaml}
    #scatter: "#grep.infile"
    inputs:
      - id: "#grep.pattern"
        source: "#mutation"
      - id: "#grep.infile"
        source: "#tumorin"
    outputs:
      - id: "#greptumor.outfile"
  - id: "#grepnormal"
    run: {import: grep.cwl.yaml}
    #scatter: "#grep.infile"
    inputs:
      - id: "#grep.pattern"
        source: "#mutation"
      - id: "#grep.infile"
        source: "#normalin"
    outputs:
      - id: "#grepnormal.outfile"
  - id: "#wc"
    run: {import: wc.cwl.yaml}
    inputs:
      - id: "#wc.infile"
        source: ["#grepnormal.outfile",  "#greptumor.outfile"]
    outputs:
      - id: "#wc.outfile"

This does not work, when I try to have it work on multiple normal/tumor input files into grep scattered:

#!/usr/bin/env cwl-runner
class: Workflow
requirements:
  - class: ScatterFeatureRequirement
inputs:
  - id: "#mutation"
    type: string
  - id: "#normalin"
    type: {type: array, items: File}
  - id: "#tumorin"
    type: {type: array, items: File}
outputs:
  - id: "#outfile"
    type: File
    source: "#wc.outfile"
steps:
  - id: "#greptumor"
    run: {import: grep.cwl.yaml}
    scatter: "#grep.infile"
    inputs:
      - id: "#grep.pattern"
        source: "#mutation"
      - id: "#grep.infile"
        source: "#tumorin"
    outputs:
      - id: "#greptumor.outfile"
  - id: "#grepnormal"
    run: {import: grep.cwl.yaml}
    scatter: "#grep.infile"
    inputs:
      - id: "#grep.pattern"
        source: "#mutation"
      - id: "#grep.infile"
        source: "#normalin"
    outputs:
      - id: "#grepnormal.outfile"
  - id: "#wc"
    run: {import: wc.cwl.yaml}
    inputs:
      - id: "#wc.infile"
        source: ["#grepnormal.outfile",  "#greptumor.outfile"]
    outputs:
      - id: "#wc.outfile"

Interestingly though, this small change does work, when you only pass one of the scatter/gathered inputs into the final wc.

#!/usr/bin/env cwl-runner
class: Workflow
requirements:
  - class: ScatterFeatureRequirement
inputs:
  - id: "#mutation"
    type: string
  - id: "#normalin"
    type: {type: array, items: File}
  - id: "#tumorin"
    type: {type: array, items: File}
outputs:
  - id: "#outfile"
    type: File
    source: "#wc.outfile"
steps:
  - id: "#greptumor"
    run: {import: grep.cwl.yaml}
    scatter: "#grep.infile"
    inputs:
      - id: "#grep.pattern"
        source: "#mutation"
      - id: "#grep.infile"
        source: "#tumorin"
    outputs:
      - id: "#greptumor.outfile"
  - id: "#grepnormal"
    run: {import: grep.cwl.yaml}
    scatter: "#grep.infile"
    inputs:
      - id: "#grep.pattern"
        source: "#mutation"
      - id: "#grep.infile"
        source: "#normalin"
    outputs:
      - id: "#grepnormal.outfile"
  - id: "#wc"
    run: {import: wc.cwl.yaml}
    inputs:
      - id: "#wc.infile"
        source: ["#grepnormal.outfile"]
    outputs:
      - id: "#wc.outfile"

So it seems to me that this is an issue only when you use the program accepting an array of inputs and you want to pass two of those scatter/gathered things into a final output. Maybe the data structure being output in the scatter/gather is weird when you put it into a list? The error is this about a dict being expected:

> cwltool mutationfinder3.cwl.yaml --mutation GCATCCA --normalin normal.fastq --tumorin tumor.fastq 
/anaconda/bin/cwltool 1.0.20151026181844
[job 4386765520] /var/folders/8r/g00jq11j2yb586cz04tfpydc0000gn/T/tmpm8fJ5E$ grep GCATCCA /Users/john/workspaces/commonwl_examples/t790m_detector/normal.fastq > /var/folders/8r/g00jq11j2yb586cz04tfpydc0000gn/T/tmpm8fJ5E/out.txt
[job 4386765840] /var/folders/8r/g00jq11j2yb586cz04tfpydc0000gn/T/tmpUclOsb$ grep GCATCCA /Users/john/workspaces/commonwl_examples/t790m_detector/tumor.fastq > /var/folders/8r/g00jq11j2yb586cz04tfpydc0000gn/T/tmpUclOsb/out.txt
Workflow error:
  Error validating input record, could not validate field `infile` because
  At position 0
    `[{'checksum': 'sha1$e62341a47da9cb155ec0f531b525a97d2bd86455',
      'class': 'File',
      'path': '/var/folders/8r/g00jq11j2yb586cz04tfpydc0000gn/T/tmpm8fJ5E/out.txt'[...]`
     is not a dict