Using --cachedir option leaves output of workflow in cache instead of the directory the workflow is run

[prismofeverything]

The title basically sums it up. When running cwl-runner and specifying a --cachedir, the output that was previously placed in the directory the workflow is run is instead left nested in the cache.

Shouldn't the use of --cachedir be transparent to the function of the tool, and leave the "interface" of the tool the same? IE, the output should go to the same place whether you are using --cachedir or not.

[thomasyu888]

+1

[prismofeverything]

Any word on this? It is currently a blocker for being able to use this feature.

[tetron]

Would you prefer the files in the output directory be symlinked, hard linked (if possible) copied, or moved? (the last one would prevent reuse of any steps that contribute files to the workflow output).

[prismofeverything]

Copied would be ideal. That would keep the behavior identical whether you
are using the cache or not, which is really the point. The use of the cache
should be invisible to other tools at either end of the pipeline.

On Mon, May 23, 2016 at 4:20 PM, Peter Amstutz notifications@github.com
wrote:

Would you prefer the files in the output directory be symlinked, hard
linked (if possible) copied, or moved? (the last one would prevent reuse of
any steps that contribute files to the workflow output).

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#86 (comment)

[tetron]

Take a look at cwltool.workflow.WorkflowJob.job() and look for if move_outputs: this is the code that moves files from the intermediate output directory to the final output directory. It should be easy to add a code path that instead copies the files when cachedir is in kwargs.

[tetron]

https://github.com/common-workflow-language/cwltool/blob/master/cwltool/workflow.py#L334

[kapilkd13]

Hi @tetron, Is this issue resolved. Also I couldn't find the move_outputs: related code in cwltool.workflow.WorkflowJob.job(). I guess that functionality has been moved to https://github.com/common-workflow-language/cwltool/blob/master/cwltool/process.py#L256 ,is it correct.