Skip to content

Commit e62e5b3

Browse files
GlassOfWhiskeyGlassOfWhiskey
committed
Added loop support
1 parent b84ec22 commit e62e5b3

23 files changed

+881
-5
lines changed

‎.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# JetBrains
2+
.idea/

‎Workflow.yml

Lines changed: 107 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -445,6 +445,56 @@ $graph:
445445
to connect the output value to downstream parameters.
446446
447447
448+
- name: LoopInput
449+
type: record
450+
extends: [Identified, Sink]
451+
fields:
452+
- name: source
453+
doc: |
454+
Specifies one or more of the step output parameters that will
455+
provide input to the loop iterations after the first one (inputs
456+
of the first iteration are the step input parameters).
457+
jsonldPredicate:
458+
"_id": "cwl:source"
459+
"_type": "@id"
460+
refScope: 2
461+
type:
462+
- string?
463+
- string[]?
464+
- name: default
465+
type: ["null", File, Directory, Any]
466+
doc: |
467+
The default value for this parameter to use if either there is no
468+
`source` field, or the value produced by the `source` is `null`. The
469+
default must be applied prior to scattering or evaluating `valueFrom`.
470+
jsonldPredicate:
471+
_id: "sld:default"
472+
noLinkCheck: true
473+
- name: valueFrom
474+
type:
475+
- "null"
476+
- string
477+
- Expression
478+
jsonldPredicate: "cwl:valueFrom"
479+
doc: |
480+
To use valueFrom, [StepInputExpressionRequirement](#StepInputExpressionRequirement) must
481+
be specified in the workflow or workflow step requirements.
482+
483+
If `valueFrom` is a constant string value, use this as the value for
484+
this input parameter.
485+
486+
If `valueFrom` is a parameter reference or expression, it must be
487+
evaluated to yield the actual value to be assigned to the input field.
488+
489+
The `self` value in the parameter reference or expression must be
490+
`null` if there is no `source` field, or the value of the
491+
parameter(s) specified in the `source` field.
492+
493+
The value of `inputs` in the parameter reference or expression must be
494+
the input object to the previous iteration of the workflow step (or the initial
495+
inputs for the first iteration).
496+
497+
448498
- name: ScatterMethod
449499
type: enum
450500
docParent: "#WorkflowStep"
@@ -455,6 +505,15 @@ $graph:
455505
- flat_crossproduct
456506

457507

508+
- name: LoopOutputMethod
509+
type: enum
510+
docParent: "#WorkflowStep"
511+
doc: The loop output method, as described in [workflow step loop](#WorkflowStep).
512+
symbols:
513+
- last
514+
- all
515+
516+
458517
- name: WorkflowStep
459518
type: record
460519
extends: [Identified, Labeled, sld:Documented]
@@ -507,7 +566,7 @@ $graph:
507566
output arrays must be flattened to a single level, but otherwise listed in the
508567
order that the input arrays are listed in the `scatter` field.
509568
510-
# Conditional execution (Optional)
569+
# Conditional and iterative execution (Optional)
511570
512571
Conditional execution makes execution of a step conditional on an
513572
expression. A step that is not executed is "skipped". A skipped
@@ -524,12 +583,37 @@ $graph:
524583
input object (or individual scatter job), and returns a boolean
525584
value. It is an error if this expression returns a value other
526585
than `true` or `false`.
527-
528-
Conditionals in CWL are an optional feature and are not required
529-
to be implemented by all consumers of CWL documents. An
530-
implementation that does not support conditionals must return a
586+
587+
The `loop` field controls iterative execution. It defines the input
588+
parameters of the loop iterations after the first one (inputs of the
589+
first iteration are the step input parameters, as usual). If no
590+
`loop` rule is specified for a given step `in` field, the initial
591+
value is kept constant among all iterations.
592+
593+
When a `loop` field is present, the `when` field is mandatory. It is
594+
evaluated before each loop iteration and acts as a termination condition:
595+
as soon as the `when` expression evaluates to `false`, the loop terminates
596+
and the step outputs are propagated to the subsequent workflow steps.
597+
598+
The `outputMethod` field describes how to deal with loop outputs after
599+
termination:
600+
601+
* **last** specifies that only the last computed element for each output
602+
parameter should be propagated to the subsequenct steps. This is the
603+
default value.
604+
605+
* **all** specifies that a single ordered array with all output values
606+
computed at the end of each loop iteration should be propagated to the
607+
subsequent steps.
608+
609+
Conditionals and iterative execution in CWL are an optional features
610+
and are not required to be implemented by all consumers of CWL documents.
611+
An implementation that does not support conditionals must return a
531612
fatal error when attempting to execute a workflow that uses
532613
conditional constructs the implementation does not support.
614+
615+
At this time, the `loop` field is not compatible with the `scatter` field.
616+
Combining the two in the same step will produce an error.
533617
534618
# Subworkflows
535619
@@ -621,6 +705,24 @@ $graph:
621705
jsonldPredicate:
622706
"_id": "cwl:scatterMethod"
623707
"_type": "@vocab"
708+
- name: loop
709+
doc: |
710+
Defines the input parameters of the loop iterations after the first one
711+
(inputs of the first iteration are the step input parameters). If no
712+
`loop` rule is specified for a given step `in` field, the initial value
713+
is kept constant among all iterations.
714+
type: LoopInput[]?
715+
jsonldPredicate:
716+
_id: "cwl:loop"
717+
mapSubject: id
718+
mapPredicate: source
719+
- name: outputMethod
720+
doc: |
721+
Required if `loop` is defined.
722+
type: LoopOutputMethod?
723+
jsonldPredicate:
724+
"_id": "cwl:outputMethod"
725+
"_type": "@vocab"
624726

625727

626728
- name: Workflow

‎conformance_tests.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3392,3 +3392,5 @@
33923392
}
33933393
}
33943394
tags: [ required, command_line_tool ]
3395+
3396+
- $import: tests/loop/test-index.yaml

‎tests/loop/all-output-loop-no-iteration.cwl

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
#!/usr/bin/env cwl-runner
2+
cwlVersion: v1.3.0-dev1
3+
class: Workflow
4+
requirements:
5+
InlineJavascriptRequirement: {}
6+
inputs:
7+
i1: int
8+
outputs:
9+
o1:
10+
type: int[]
11+
outputSource: subworkflow/o1
12+
steps:
13+
subworkflow:
14+
when: $(inputs.i1 < 1)
15+
loop:
16+
i1: o1
17+
outputMethod: all
18+
run:
19+
class: ExpressionTool
20+
inputs:
21+
i1: int
22+
outputs:
23+
o1: int
24+
expression: >
25+
${return {'o1': inputs.i1 + 1};}
26+
in:
27+
i1: i1
28+
out: [o1]

‎tests/loop/all-output-loop.cwl

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
#!/usr/bin/env cwl-runner
2+
cwlVersion: v1.3.0-dev1
3+
class: Workflow
4+
requirements:
5+
InlineJavascriptRequirement: {}
6+
inputs:
7+
i1: int
8+
outputs:
9+
o1:
10+
type: int[]
11+
outputSource: subworkflow/o1
12+
steps:
13+
subworkflow:
14+
when: $(inputs.i1 < 10)
15+
loop:
16+
i1: o1
17+
outputMethod: all
18+
run:
19+
class: ExpressionTool
20+
inputs:
21+
i1: int
22+
outputs:
23+
o1: int
24+
expression: >
25+
${return {'o1': inputs.i1 + 1};}
26+
in:
27+
i1: i1
28+
out: [o1]

‎tests/loop/default-value-loop.cwl

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
#!/usr/bin/env cwl-runner
2+
cwlVersion: v1.3.0-dev1
3+
class: Workflow
4+
requirements:
5+
InlineJavascriptRequirement: {}
6+
ScatterFeatureRequirement: {}
7+
SubworkflowFeatureRequirement: {}
8+
inputs:
9+
i1: int
10+
outputs:
11+
o1:
12+
type: int[]
13+
outputSource: loop/o1
14+
pickValue: all_non_null
15+
steps:
16+
loop:
17+
when: $(inputs.i1 < 20)
18+
loop:
19+
i1:
20+
source: o1
21+
default: 5
22+
outputMethod: all
23+
run:
24+
class: Workflow
25+
inputs:
26+
i1: int
27+
outputs:
28+
o1:
29+
type: int?
30+
outputSource: big_values/o1
31+
steps:
32+
big_values:
33+
when: $(inputs.i1 >= 5)
34+
run:
35+
class: ExpressionTool
36+
inputs:
37+
i1: int
38+
outputs:
39+
o1: int
40+
expression: >
41+
${return {'o1': inputs.i1 + 3};}
42+
in:
43+
i1: i1
44+
out: [ o1 ]
45+
in:
46+
i1: i1
47+
out: [ o1 ]

‎tests/loop/invalid-loop-scatter.cwl

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
#!/usr/bin/env cwl-runner
2+
cwlVersion: v1.3.0-dev1
3+
class: Workflow
4+
requirements:
5+
InlineJavascriptRequirement: {}
6+
ScatterFeatureRequirement: {}
7+
SubworkflowFeatureRequirement: {}
8+
inputs:
9+
i1: int[]
10+
i2: int
11+
outputs:
12+
o1:
13+
type: int[]
14+
outputSource: subworkflow/o1
15+
steps:
16+
subworkflow:
17+
when: $(inputs.i1 < 10)
18+
loop:
19+
i1: o1
20+
outputMethod: last
21+
run:
22+
class: ExpressionTool
23+
inputs:
24+
i1: int
25+
i2: int
26+
outputs:
27+
o1: int
28+
expression: >
29+
${return {'o1': inputs.i1 + inputs.i2};}
30+
in:
31+
i1: i1
32+
i2: i2
33+
scatter: i1
34+
out: [o1]

‎tests/loop/invalid-multi-source-loop-no-requirement.cwl

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
#!/usr/bin/env cwl-runner
2+
cwlVersion: v1.3.0-dev1
3+
class: Workflow
4+
requirements:
5+
InlineJavascriptRequirement: {}
6+
ScatterFeatureRequirement: {}
7+
SubworkflowFeatureRequirement: {}
8+
inputs:
9+
i1: int
10+
outputs:
11+
o1:
12+
type: int[]
13+
outputSource: [loop/osmall, loop/obig]
14+
linkMerge: merge_flattened
15+
pickValue: all_non_null
16+
steps:
17+
loop:
18+
when: $(inputs.i1 < 20)
19+
loop:
20+
i1:
21+
source: [ osmall, obig ]
22+
pickValue: the_only_non_null
23+
outputMethod: all
24+
run:
25+
class: Workflow
26+
inputs:
27+
i1: int
28+
outputs:
29+
osmall:
30+
type: int?
31+
outputSource: small_values/o1
32+
obig:
33+
type: int?
34+
outputSource: big_values/o1
35+
steps:
36+
small_values:
37+
when: $(inputs.i1 < 5)
38+
run:
39+
class: ExpressionTool
40+
inputs:
41+
i1: int
42+
outputs:
43+
o1: int
44+
expression: >
45+
${return {'o1': inputs.i1 + 1};}
46+
in:
47+
i1: i1
48+
out: [o1]
49+
big_values:
50+
when: $(inputs.i1 >= 5)
51+
run:
52+
class: ExpressionTool
53+
inputs:
54+
i1: int
55+
outputs:
56+
o1: int
57+
expression: >
58+
${return {'o1': inputs.i1 + 3};}
59+
in:
60+
i1: i1
61+
out: [ o1 ]
62+
in:
63+
i1: i1
64+
out: [osmall, obig]

‎tests/loop/invalid-no-when.cwl

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
#!/usr/bin/env cwl-runner
2+
cwlVersion: v1.3.0-dev1
3+
class: Workflow
4+
requirements:
5+
InlineJavascriptRequirement: {}
6+
ScatterFeatureRequirement: {}
7+
SubworkflowFeatureRequirement: {}
8+
inputs:
9+
i1: int
10+
i2: int
11+
outputs:
12+
o1:
13+
type: int
14+
outputSource: subworkflow/o1
15+
steps:
16+
subworkflow:
17+
loop:
18+
i1: o1
19+
outputMethod: last
20+
run:
21+
class: ExpressionTool
22+
inputs:
23+
i1: int
24+
i2: int
25+
outputs:
26+
o1: int
27+
expression: >
28+
${return {'o1': inputs.i1 + inputs.i2};}
29+
in:
30+
i1: i1
31+
i2: i2
32+
out: [o1]

0 commit comments

Comments
 (0)