Description
Long story short:
It appears as though cwltool validates that both the default and job_order paths exist.
Ideally, it would only validate the job_order paths if provided, the default paths if not provided.
Long story:
A toy command and file tree to demonstrate this follows
Required files: https://gist.github.com/denis-yuen/a1d36345ecc4bdc1580a
.
├── 8e888694-9c56-4529-a750-d6bfbd4a74e7.txt -> inputs/0bf6ab9f-83a8-4b72-ad67-460eb696bd64/ref_file_2
├── b4cdad91-676a-446c-a635-57453f17617a.txt -> inputs/d2816fb8-7773-4b2a-bd31-ee4a7efa8e04/ref_file_1
├── configs
├── foo.json
├── image-descriptor.cwl
├── inputs
│ ├── 0bf6ab9f-83a8-4b72-ad67-460eb696bd64
│ │ └── ref_file_2
│ ├── 0e129840-fa77-4cec-a885-19b1e538507a
│ │ └── hello_input
│ └── d2816fb8-7773-4b2a-bd31-ee4a7efa8e04
│ └── ref_file_1
├── logs
├── node-engine.cwl
├── outputs
│ └── hello-output.txt
└── working
When I run the following command, it executes successfully. ( cwltool --outdir /home/dyuen/consonance/consonance-arch/./datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995/outputs/ image-descriptor.cwl foo.json )
However, note the symbolic links in the root. These correspond to the default paths in image-descriptor.cwl which should be overridden by the entries in foo.json.
But when I delete those two links, cwltool dies with the following validation error.
/usr/local/bin/cwltool 1.0.20151013173827
Tool definition failed validation:
While checking field `inputs`
While checking object `file:///home/dyuen/consonance/consonance-arch/datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995/image-descriptor.cwl#ref_file_1`
While checking field `default`
Field `path` contains undefined reference to `file:///home/dyuen/consonance/consonance-arch/datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995/8e888694-9c56-4529-a750-d6bfbd4a74e7.txt`
While checking object `file:///home/dyuen/consonance/consonance-arch/datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995/image-descriptor.cwl#ref_file_2`
While checking field `default`
Field `path` contains undefined reference to `file:///home/dyuen/consonance/consonance-arch/datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995/b4cdad91-676a-446c-a635-57453f17617a.txt`
Note that its validating the default paths, not the paths provided in foo.json. However, if I move the files referenced in the foo.json to match the defaults.
.
├── 8e888694-9c56-4529-a750-d6bfbd4a74e7.txt
├── b4cdad91-676a-446c-a635-57453f17617a.txt
├── configs
├── foo.json
├── hello-output.txt
├── image-descriptor.cwl
├── inputs
│ └── 0e129840-fa77-4cec-a885-19b1e538507a
│ └── hello_input
├── logs
├── node-engine.cwl
├── outputs
└── working
Then it dies saying it can't find the paths in the foo.json
dyuen@odl-dyuen:~/consonance/consonance-arch/datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995$ cwltool --outdir /home/dyuen/consonance/consonance-arch/./datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995/outputs/ image-descriptor.cwl foo.json
/usr/local/bin/cwltool 1.0.20151013173827
Unknown hint file:///home/dyuen/consonance/consonance-arch/datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995/ResourceRequirement
Got workflow error
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/cwltool/main.py", line 153, in single_job_executor
for r in jobiter:
File "/usr/local/lib/python2.7/dist-packages/cwltool/draft2tool.py", line 127, in job
builder.pathmapper = self.makePathMapper(reffiles, input_basedir, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/cwltool/draft2tool.py", line 62, in makePathMapper
return DockerPathMapper(reffiles, input_basedir)
File "/usr/local/lib/python2.7/dist-packages/cwltool/pathmapper.py", line 73, in __init__
st = os.lstat(deref)
OSError: [Errno 2] No such file or directory: '/home/dyuen/consonance/consonance-arch/./datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995/inputs/0bf6ab9f-83a8-4b72-ad67-460eb696bd64/ref_file_2'
Workflow error:
[Errno 2] No such file or directory: '/home/dyuen/consonance/consonance-arch/./datastore/launcher-03ad4441-c074-4893-b5bd-6d14aabb4995/inputs/0bf6ab9f-83a8-4b72-ad67-460eb696bd64/ref_file_2'