CWL 1.2 support: booleans, when clause, input mappings not working

[miguelcorreia19]

Hi team,

I’ve noticed that the current CWL support in zoo-project-dru appears to be based on version 1.0. Are there any plans to upgrade support to CWL v1.2?

As far as I can tell, the following features are not currently supported:

• boolean inputs
• optional steps using the when clause
• some input mappings don’t seem to be parsed correctly

I’d really appreciate your input on this. Please let me know if sharing a test case would be helpful.

Thanks!

[gfenoy]

Hi @miguelcorreia19,
Thanks for your feedback.

The boolean input handling during the execution should now be fixed with the following commit: EOEPCA/zoo-calrissian-runner@0aa701c.

To test it locally, I would like you to provide an example of the when clause.

The same remark applies to the input mappings. Which ones are not parsed correctly?

Having a test case at hand would help.

[miguelcorreia19]

Hi @gfenoy,

I've created the following three workflows. The job fails with these, but runs with cwltool.

1) Optional steps - hello_world_0_2_2.cwl

• step1: generates a hello world file
• step2: optional step, triggered based on input parameters
• step3: generates a sample STAC directory

workflow location: https://pipe-drive.inesctec.pt/application-packages/workflows/hello-world/hello_world_0_2_2.cwl

running with cwltool:

cwltool https://pipe-drive.inesctec.pt/application-packages/workflows/hello-world/hello_world_0_2_2.cwl#helloworld_pipeline --name MyName

input:

{
  "inputs": {
    "name": "MyName"
  }
}

job message:

"Unable to run the Service. The message returned back by the Service was the following: Exception during execution...\nTraceback (most recent call last):\n  File \"/opt/zooservices_user/inesctec/helloworld_pipeline/service.py\", line 311, in helloworld_pipeline\n    exit_status = runner.execute()\n  File \"/usr/local/lib/python3.8/dist-packages/zoo_calrissian_runner/__init__.py\", line 398, in execute\n    if not (self.assert_parameters()):\n  File \"/usr/local/lib/python3.8/dist-packages/zoo_calrissian_runner/__init__.py\", line 389, in assert_parameters\n    return all(\n  File \"/usr/local/lib/python3.8/dist-packages/zoo_calrissian_runner/__init__.py\", line 390, in <genexpr>\n    elem in list(self.get_processing_parameters().keys())\n  File \"/usr/local/lib/python3.8/dist-packages/zoo_calrissian_runner/__init__.py\", line 381, in get_processing_parameters\n    return self.inputs.get_processing_parameters()\n  File \"/usr/local/lib/python3.8/dist-packages/zoo_calrissian_runner/__init__.py\", line 237, in get_processing_parameters\n    res[key]=int(value[\"value\"])\nValueError: invalid literal for int() with base 10: 'NULL'\n\n"

2) Optional inputs - hello_world_0_2_3.cwl

• step1: generates a STAC directory

workflow location: https://pipe-drive.inesctec.pt/application-packages/workflows/hello-world/hello_world_0_2_3.cwl

running with cwltool:

cwltool https://pipe-drive.inesctec.pt/application-packages/workflows/hello-world/hello_world_0_2_3.cwl#helloworld_pipeline

input:

{
  "inputs": {}
}

job message:

"Unable to run the Service. The message returned back by the Service was the following: Exception during execution...\nTraceback (most recent call last):\n  File \"/opt/zooservices_user/inesctec/helloworld_pipeline/service.py\", line 295, in helloworld_pipeline\n    runner = ZooCalrissianRunner(\n  File \"/usr/local/lib/python3.8/dist-packages/zoo_calrissian_runner/__init__.py\", line 292, in __init__\n    self.inputs = ZooInputs(inputs)\n  File \"/usr/local/lib/python3.8/dist-packages/zoo_calrissian_runner/__init__.py\", line 205, in __init__\n    for inp in inputs:\nTypeError: 'NoneType' object is not iterable\n\n",

This happens when no input is provided.

3) ExpressionTool volume issue - hello_world_0_2_0.cwl

• step1: returns a hello world file
• step2: ExpressionTool simulating a json file
• step3: transform inputs in a STAC directory

workflow location: https://pipe-drive.inesctec.pt/application-packages/workflows/hello-world/hello_world_0_2_0.cwl

running with cwltool:

cwltool https://pipe-drive.inesctec.pt/application-packages/workflows/hello-world/hello_world_0_2_0.cwl#helloworld_pipeline --name MyName

input:

{
  "inputs": {
    "name": "MyName"
  }
}

pod log:

ERROR Workflow error:
Could not find a persistent volume mounted for /tmp/rw78cuj_/metadata.json
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/cwltool/main.py", line 1295, in main
    tool, initialized_job_order_object, runtimeContext, logger=_logger
  File "/usr/local/lib/python3.7/site-packages/cwltool/executors.py", line 59, in __call__
    return self.execute(process, job_order_object, runtime_context, logger)
  File "/usr/local/lib/python3.7/site-packages/cwltool/executors.py", line 138, in execute
    self.run_jobs(process, job_order_object, logger, runtime_context)
  File "/usr/local/lib/python3.7/site-packages/calrissian/executor.py", line 422, in run_jobs
    futures = self.enqueue_jobs_from_iterator(job_iterator, logger, runtime_context, pool_executor)
  File "/usr/local/lib/python3.7/site-packages/calrissian/executor.py", line 380, in enqueue_jobs_from_iterator
    self.raise_if_exception_queued(futures, logger)
  File "/usr/local/lib/python3.7/site-packages/calrissian/executor.py", line 260, in raise_if_exception_queued
    raise exceptions[0]
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/calrissian/job.py", line 667, in run
    pod = self.create_kubernetes_runtime(runtimeContext) # analogous to create_runtime()
  File "/usr/local/lib/python3.7/site-packages/calrissian/job.py", line 528, in create_kubernetes_runtime
    any_path_okay=True)
  File "/usr/local/lib/python3.7/site-packages/cwltool/job.py", line 719, in add_volumes
    runtime, vol, host_outdir_tgt, secret_store, tmpdir_prefix
  File "/usr/local/lib/python3.7/site-packages/cwltool/job.py", line 684, in create_file_and_add_volume
    self.append_volume(runtime, new_file, volume.target, writable=writable)
  File "/usr/local/lib/python3.7/site-packages/calrissian/job.py", line 695, in append_volume
    self._add_volume_binding(source, target, writable)
  File "/usr/local/lib/python3.7/site-packages/calrissian/job.py", line 573, in _add_volume_binding
    self.volume_builder.add_volume_binding(source, target, writable)
  File "/usr/local/lib/python3.7/site-packages/calrissian/job.py", line 168, in add_volume_binding
    raise VolumeBuilderException('Could not find a persistent volume mounted for {}'.format(source))
calrissian.job.VolumeBuilderException: Could not find a persistent volume mounted for /tmp/rw78cuj_/metadata.json

[gfenoy]

Hi @miguelcorreia19

Thanks a lot for your feedback.

1) Optional steps - hello_world_0_2_2.cwl

You can use this CWL from the ZOO-Project-DRU with the following merged PR.

2) Optional inputs - hello_world_0_2_3.cwl

Same remark as before, should work with the latest zoo-calrissian-runner develop branch.

3) ExpressionTool volume issue - hello_world_0_2_0.cwl

I am still working on debugging this one and hope to return soon with a fix.

The easiest way to test this update in zoo-calrissian-runner before a new chart is published is to use the skaffold dev command from the ZOO-Project repository. This will build a fresh ZOO-Project-DRU and deploy this specific version.

[fabricebrito]

@gfenoy I think this workflow https://github.com/eoap/how-to/blob/main/cwl-workflows/conditional-workflows.cwl helps testing the fix

[miguelcorreia19]

Hi Gerald,

I found two more issues:

4) Optional inputs with multiple types hello_world_0_3_1

• step1: generates a hello world file
• step3: generates a sample STAC directory

workflow location: https://pipe-drive.inesctec.pt/application-packages/workflows/hello-world/hello_world_0_3_1.cwl

running with cwltool:

cwltool https://pipe-drive.inesctec.pt/application-packages/workflows/hello-world/hello_world_0_3_1.cwl#helloworld_pipeline --name MyName

input:

{
  "inputs": {
    "name": "MyName"
  }
}

I didn't pass the optional input something.

job message:

The <something> argument was not specified in DataInputs but is required according to the ZOO ServicesProvider configuration file.

5) Reserved input names hello_world_0_3_2

• step1: generates a hello world file

running with cwltool:

cwltool https://pipe-drive.inesctec.pt/application-packages/workflows/hello-world/hello_world_0_3_2.cwl#helloworld_pipeline --s3_bucket MyName

input:

{
  "inputs": {
    "s3_bucket": "MyName"
  }
}

The file result should contain the text Hello MyName from python 3.6.6!. However, the actual content is Hello result from python 3.6.6!. This is because the input s3_bucket is "reserved by the service" and is replaced by the defined k8s env vars (workflow.additionalInputs). Can this behaviour be changed?

[gfenoy]

Hi Miguel,
My comments are inline below.

4) Optional inputs with multiple types hello_world_0_3_1

The following definition is not supported for inputs:

      something:
        type:
          - string?
          - File?

In the ZOO-Project server implementation, defining an input as ComplexData or LiteralData is impossible.
You have to choose between File? (ComplexData) and string? (LiteralData).

5) Reserved input names hello_world_0_3_2

What Helm chart version are you currently using for your tests? What is the service template in use?
I tried deploying and executing the workflow using Helm chart version 0.4.17; result.txt contains the expected string.

[gfenoy]

3) ExpressionTool volume issue - hello_world_0_2_0.cwl

• step1: returns a hello world file
• step2: ExpressionTool simulating a json file
• step3: transform inputs in a STAC directory

workflow location: https://pipe-drive.inesctec.pt/application-packages/workflows/hello-world/hello_world_0_2_0.cwl

running with cwltool:

cwltool https://pipe-drive.inesctec.pt/application-packages/workflows/hello-world/hello_world_0_2_0.cwl#helloworld_pipeline --name MyName
input:

{
"inputs": {
"name": "MyName"
}
}
pod log:

ERROR Workflow error:
Could not find a persistent volume mounted for /tmp/rw78cuj_/metadata.json
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/cwltool/main.py", line 1295, in main
    tool, initialized_job_order_object, runtimeContext, logger=_logger
  File "/usr/local/lib/python3.7/site-packages/cwltool/executors.py", line 59, in __call__
    return self.execute(process, job_order_object, runtime_context, logger)
  File "/usr/local/lib/python3.7/site-packages/cwltool/executors.py", line 138, in execute
    self.run_jobs(process, job_order_object, logger, runtime_context)
  File "/usr/local/lib/python3.7/site-packages/calrissian/executor.py", line 422, in run_jobs
    futures = self.enqueue_jobs_from_iterator(job_iterator, logger, runtime_context, pool_executor)
  File "/usr/local/lib/python3.7/site-packages/calrissian/executor.py", line 380, in enqueue_jobs_from_iterator
    self.raise_if_exception_queued(futures, logger)
  File "/usr/local/lib/python3.7/site-packages/calrissian/executor.py", line 260, in raise_if_exception_queued
    raise exceptions[0]
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/calrissian/job.py", line 667, in run
    pod = self.create_kubernetes_runtime(runtimeContext) # analogous to create_runtime()
  File "/usr/local/lib/python3.7/site-packages/calrissian/job.py", line 528, in create_kubernetes_runtime
    any_path_okay=True)
  File "/usr/local/lib/python3.7/site-packages/cwltool/job.py", line 719, in add_volumes
    runtime, vol, host_outdir_tgt, secret_store, tmpdir_prefix
  File "/usr/local/lib/python3.7/site-packages/cwltool/job.py", line 684, in create_file_and_add_volume
    self.append_volume(runtime, new_file, volume.target, writable=writable)
  File "/usr/local/lib/python3.7/site-packages/calrissian/job.py", line 695, in append_volume
    self._add_volume_binding(source, target, writable)
  File "/usr/local/lib/python3.7/site-packages/calrissian/job.py", line 573, in _add_volume_binding
    self.volume_builder.add_volume_binding(source, target, writable)
  File "/usr/local/lib/python3.7/site-packages/calrissian/job.py", line 168, in add_volume_binding
    raise VolumeBuilderException('Could not find a persistent volume mounted for {}'.format(source))
calrissian.job.VolumeBuilderException: Could not find a persistent volume mounted for /tmp/rw78cuj_/metadata.json

We are having the same issue using ExpressionTool. However, we have solutions for the ExpressionTool named simulate_metadata_2stac in your example.

Solution 1

You can replace it with the following CommandLine to get the exact same result.

  - class: CommandLineTool
    id: simulate_metadata_2stac
    baseCommand:
      - cat
      - metadata-in.json
    requirements:
      DockerRequirement:
        dockerPull: alpine:3.14
      InitialWorkDirRequirement:
        listing:
          - entryname: metadata-in.json
            entry: |
              {
                "description": "This is an example metadata file for 2stac2.",
                "keywords": ["example", "metadata", "helloworld"],
                "bbox": [-180, -90, 180, 90],
                "media_type": "application/json",
                "geometry": {
                  "type": "Polygon",
                  "coordinates": [
                    [
                      [-180, -90],
                      [-180, 90],
                      [180, 90],
                      [180, -90],
                      [-180, -90]
                    ]
                  ]
                }
              }
    inputs: []
    outputs:
      metadata:
        type: File
        format: edam:format_3464
        outputBinding:
          glob: metadata.json
    stdout: metadata.json

Solution 2

You can use the CWL below, which embeds the ExpressionTool as InitialWorkdirRequirement.listing. This version does not add the simulate_metadata_2stac step.

cwlVersion: v1.2
$namespaces:
  s: https://schema.org/
  edam: http://edamontology.org/
$graph:
  - class: Workflow
    id: helloworld_pipeline02
    inputs:
      s3_bucket:
        type: string
        doc: Some keyword
    steps:
      step_helloworld:
        run: '#helloworld'
        in:
          name: s3_bucket
        out:
          - result
      step_2stac:
        run: '#2stac2_simulation'
        in:
          result: step_helloworld/result
        out:
          - stac_result
    outputs:
      - id: wf_outputs
        outputSource:
          - step_2stac/stac_result
        type: Directory
    requirements:
      InlineJavascriptRequirement: {}
    s:name: helloworld_pipeline
    s:description: |
      This pipeline runs the helloworld tool and returns the result.
    s:keywords:
      - helloworld
    s:softwareVersion: 0.3.2
    s:producer:
      class: s:Organization
      s:name: INESCTEC
      s:url: https://inesctec.pt
      s:address:
        class: s:PostalAddress
        s:addressCountry: PT
    s:sourceOrganization:
      - class: s:Organization
        s:name: INESCTEC
        s:url: https://inesctec.pt
        s:address:
          class: s:PostalAddress
          s:addressCountry: PT
    s:author:
      - class: s:Person
        s:name: Miguel Correia
        s:email: miguel.r.correia@inesctec.pt
    s:codeRepository: >-
      https://pipe-drive.inesctec.pt/application-packages/workflows/hello-world/hello_world_0_3_2.cwl
    s:dateCreated: '2025-06-11T20:09:26Z'
  - class: CommandLineTool
    id: helloworld
    baseCommand: python
    arguments:
      - /opt/helloworld.py
      - valueFrom: $( inputs.name )
    inputs:
      name:
        type: string
        doc: Some keyword
    outputs:
      result:
        type: File
        outputBinding:
          glob: result.txt
        doc: result file
    requirements:
      ResourceRequirement: {}
      InlineJavascriptRequirement: {}
      DockerRequirement:
        dockerPull: iliad-repository.inesctec.pt/hello-world:0.1.0
    s:name: helloworld
    s:description: Hello World example
    s:keywords:
      - helloworld
      - example
      - simple
    s:programmingLanguage: python
    s:softwareVersion: 0.1.0
    s:producer:
      class: s:Organization
      s:name: INESCTEC
      s:url: https://inesctec.pt
      s:address:
        class: s:PostalAddress
        s:addressCountry: PT
    s:sourceOrganization:
      - class: s:Organization
        s:name: INESCTEC
        s:url: https://inesctec.pt
        s:address:
          class: s:PostalAddress
          s:addressCountry: PT
    s:author:
      - class: s:Person
        s:name: Miguel Correia
        s:email: miguel.r.correia@inesctec.pt
    s:codeRepository: >-
      https://pipe-drive.inesctec.pt/application-packages/tools/hello-world/hello_world_0_1_0.cwl
    s:dateCreated: '2025-05-12T12:21:05Z'
  - class: CommandLineTool
    id: 2stac2_simulation
    baseCommand: python
    arguments:
      - /opt/2stac2.py
      - '--file'
      - valueFrom: $( inputs.result )
      - '--metadata'
      - valueFrom: $(runtime.outdir + '/metadata.json')
    inputs:
      result:
        type: File
        doc: The resulting file of the previous model to insert in STAC
    outputs:
      stac_result:
        outputBinding:
          glob: stac_result
        type: Directory
        doc: STAC output
    requirements:
      ResourceRequirement: {}
      EnvVarRequirement:
        envDef:
          PATH: >-
            /opt/conda/envs/application/bin:/opt/conda/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      InlineJavascriptRequirement: {}
      DockerRequirement:
        dockerPull: iliad-repository.inesctec.pt/2stac2:0.3.1
      InitialWorkDirRequirement:
        listing: |
          ${

            return [{
              "class": "File",
              "basename": "metadata.json",
              "contents": JSON.stringify([{
                "filename": inputs.result.basename,
                "description": "This is an example metadata file for 2stac2.",
                "keywords": ["example", "metadata", "helloworld"],
                "bbox": [-180, -90, 180, 90],
                "datetime": new Date().toISOString(),
                "media_type": "text/plain",
                "geometry": {
                  "type": "Polygon",
                  "coordinates": [
                    [
                      [-180, -90],
                      [-180, 90],
                      [180, 90],
                      [180, -90],
                      [-180, -90]
                    ]
                  ]
                },
              }])}];
          }
    s:name: 2stac2_simulation
    s:softwareVersion: 0.3.1
    s:description: Simulates a STAC result
    s:keywords:
      - stac
      - metadata
    s:programmingLanguage: python
    s:producer:
      class: s:Organization
      s:name: INESCTEC
      s:url: https://inesctec.pt
      s:address:
        class: s:PostalAddress
        s:addressCountry: PT
    s:sourceOrganization:
      - class: s:Organization
        s:name: INESCTEC
        s:url: https://inesctec.pt
        s:address:
          class: s:PostalAddress
          s:addressCountry: PT
    s:author:
      - class: s:Person
        s:name: Miguel Correia
        s:email: miguel.r.correia@inesctec.pt
    s:codeRepository: >-
      https://pipe-drive.inesctec.pt/application-packages/tools/2stac2/2stac2_simulation_0_3_1.cwl
    s:dateCreated: '2025-06-10T15:27:44Z'

We are still searching for a solution with the original ExpressionTool, cf. Duke-GCB/calrissian#193

[miguelcorreia19]

Hi @gfenoy, thanks for the updates!

About the issue "5) Reserved input names hello_world_0_3_2", it is still occurring. I tested it with the latest version (0.5.0).

I also used this cwl too: https://pipe-drive.inesctec.pt/application-packages/workflows/hello-world/hello_world_0_3_3.cwl. The system is reserving the endpoint_url property defined in the CWL as an input and replacing it with the content defined in additionalInputs instead of using the input that is being passed in.

Thanks for the alternative solutions you provided!