Problems with Directories

[karl616]

I've been trying to use Directories when that is suitable. I have a couple of issues:

Given that I define a Directory in my input file as:

dir:
  class: Directory
  location: /tmp

1. If the specified folder contains a lot of files and subfolders, cwltool (ver. 1.0.20161207161158) will spend quite some time generating "listing". Perhaps this should be done lazily?
2. cwltool crashes if you don't have permissions to access all subfolders.
3. referring to the Directory within an expression, input.dir.location == "file:///tmp" and input.dir.path does not exist. (Test code below). I would have expected that something should evaluate to /tmp? Am I missing something?

The behavior is the same if I initialize with path: /tmp instead of location. This perplexes me a bit. I don't understand why path is unset.

cwlVersion: v1.0
class: ExpressionTool

requirements:
- class: InlineJavascriptRequirement

inputs:
  dir:
    type: Directory

outputs:
  location:
    type: string
  path:
    type: string

expression: |
  ${
    return {"location":inputs.dir.location, "path": ("" + inputs.dir.path)};
  }

[tetron]

1. I agree that generating the full listing can be expensive. The plan is to change the behavior so that it only reads the full listing on request, but that is a CWL spec change so it is going to take a little longer to roll out. (In the interim we might be able to add a hint that changes the behavior for v1.0 documents.)

2. Cwltool should handle the case of inaccessible files more gracefully than crashing, but in general if you can't read all the files in a directory tree, the correct behavior is probably to fail.

3. The distinction between "location" and "path" is that "location" is a canonical URI representing a file or directory (which could be in remote storage) and "path" is where it is available on the local filesystem. Files are only guaranteed to be staged when running a CommandLineTool. ExpressionTools are executed as part of the workflow runner, which is not guaranteed to have local copies of the files being manipulated.

This complexity is present because CWL is intended to be able to describe distributed computations where the workflow runner, compute node, and storage may be three separate and distinct systems.

[karl616]

1. That is good enough for me.

2. One could also argue that the listing only contains the files you have access to. It probably comes down to the intentions with the Directory object. Me, I had an input that was a directory so I thought I specify this in the pipeline. This is probably not the way it is supposed to be used, taking the /tmp folder as a convoluted example (as it better be extracted from the runtime). Still, if I wanted to propagate this folder to a tool I'd better use a string as I cannot guarantee that I have access to everything in /tmp. My naive understanding of Directory was as a pointer to the directory rather than an object summarizing it and all sub-folders.

3. I see my mistake. The logic makes sense. Re-reading the documentation I see that this is what it says. I'm still too new to cwl to directly grok the fineries.

Thanks for taking time to straighten this out for me.