cwltool fails first iteration when it needs to pull commonworkflowlanguage/nodejs-engine

[denis-yuen]

Not sure if expected behaviour.

When cwltool needs to pull commonworkflowlanguage/nodejs-engine , it will fail running the tool. A second attempt succeeds.

ubuntu@cwl-sort-test:~/CancerCollaboratory/dockstore-tool-linux-sort$ cwltool --non-strict Dockstore.cwl test.json
/usr/local/bin/cwltool 1.0.20160108200940
Got workflow error
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/cwltool/main.py", line 158, in single_job_executor
    for r in jobiter:
  File "/usr/local/lib/python2.7/dist-packages/cwltool/draft2tool.py", line 130, in job
    j.stdout = builder.do_eval(self.tool["stdout"])
  File "/usr/local/lib/python2.7/dist-packages/cwltool/builder.py", line 165, in do_eval
    context=context, pull_image=pull_image)
  File "/usr/local/lib/python2.7/dist-packages/cwltool/expression.py", line 135, in do_eval
    return sandboxjs.interpolate(ex, jshead(r.get("expressionLib", []), rootvars))
  File "/usr/local/lib/python2.7/dist-packages/cwltool/sandboxjs.py", line 128, in interpolate
    e = execjs(scan[w[0]+1:w[1]], jslib)
  File "/usr/local/lib/python2.7/dist-packages/cwltool/sandboxjs.py", line 41, in execjs
    raise JavascriptException("Returncode was: %s\nscript was: %s\nstdout was: '%s'\nstderr was: '%s'\n" % (nodejs.returncode, script, stdoutdata, stderrdata))
JavascriptException: Returncode was: 2
script was: console.log(JSON.stringify(require("vm").runInNewContext("\"use strict\";var inputs = {\"allocatedResources\": {\"mem\": 1000, \"cpu\": 1}, \"input\": [{\"path\": \"file:///home/ubuntu/CancerCollaborat
ory/dockstore-tool-linux-sort/example.bedGraph\", \"class\": \"File\"}], \"key\": [\"1,1\", \"2,2n\"], \"output\": \"example.bedGraph.sorted\"};\nvar self = null;\nvar runtime = {\"outdirSize\": 1024, \"ram\": 102
4, \"tmpdirSize\": 1024, \"cores\": 1, \"tmpdir\": \"/tmp/job_tmp\", \"outdir\": \"/tmp/job_output\"};\n(function(){return ((inputs.output));})()", {})));

stdout was: ''
stderr was: 'Unable to find image 'commonworkflowlanguage/nodejs-engine:latest' locally
latest: Pulling from commonworkflowlanguage/nodejs-engine
f9443a9b4216: Pulling fs layer
1ddcf54c0400: Pulling fs layer
e091dad1ae93: Pulling fs layer
41f5a9faf342: Pulling fs layer
41f5a9faf342: Verifying Checksum
41f5a9faf342: Download complete
1ddcf54c0400: Verifying Checksum
1ddcf54c0400: Download complete
e091dad1ae93: Verifying Checksum
e091dad1ae93: Download complete
'

Workflow error:
  Returncode was: 2
script was: console.log(JSON.stringify(require("vm").runInNewContext("\"use strict\";var inputs = {\"allocatedResources\": {\"mem\": 1000, \"cpu\": 1}, \"input\": [{\"path\": \"file:///home/ubuntu/CancerCollaborat
ory/dockstore-tool-linux-sort/example.bedGraph\", \"class\": \"File\"}], \"key\": [\"1,1\", \"2,2n\"], \"output\": \"example.bedGraph.sorted\"};\nvar self = null;\nvar runtime = {\"outdirSize\": 1024, \"ram\": 102
4, \"tmpdirSize\": 1024, \"cores\": 1, \"tmpdir\": \"/tmp/job_tmp\", \"outdir\": \"/tmp/job_output\"};\n(function(){return ((inputs.output));})()", {})));

stdout was: ''
stderr was: 'Unable to find image 'commonworkflowlanguage/nodejs-engine:latest' locally
latest: Pulling from commonworkflowlanguage/nodejs-engine
f9443a9b4216: Pulling fs layer
1ddcf54c0400: Pulling fs layer
e091dad1ae93: Pulling fs layer
41f5a9faf342: Pulling fs layer
41f5a9faf342: Verifying Checksum
41f5a9faf342: Download complete
1ddcf54c0400: Verifying Checksum
1ddcf54c0400: Download complete
e091dad1ae93: Verifying Checksum
e091dad1ae93: Download complete
'

ubuntu@cwl-sort-test:~/CancerCollaboratory/dockstore-tool-linux-sort$ cwltool --non-strict Dockstore.cwl test.json
/usr/local/bin/cwltool 1.0.20160108200940
[job 139960086713808] /home/ubuntu/CancerCollaboratory/dockstore-tool-linux-sort$ docker run -i --volume=/home/ubuntu/CancerCollaboratory/dockstore-tool-linux-sort/example.bedGraph:/tmp/job299693805_dockstore-tool
-linux-sort/example.bedGraph:ro --volume=/home/ubuntu/CancerCollaboratory/dockstore-tool-linux-sort:/tmp/job_output:rw --volume=/tmp/tmpMKbQ3D:/tmp/job_tmp:rw --workdir=/tmp/job_output --read-only=true --user=1000
 --rm --env=TMPDIR=/tmp/job_tmp quay.io/collaboratory/dockstore-tool-linux-sort sort -k 1,1 -k 2,2n /tmp/job299693805_dockstore-tool-linux-sort/example.bedGraph > /home/ubuntu/CancerCollaboratory/dockstore-tool-li
nux-sort/example.bedGraph.sorted
Final process status is success
{
    "sorted": {
        "path": "/home/ubuntu/CancerCollaboratory/dockstore-tool-linux-sort/example.bedGraph.sorted", 
        "size": 796, 
        "class": "File", 
        "checksum": "sha1$36b0a5a18a584d095597912b702251f90db60a64"
    }
}

Workaround: pre-load the engine via docker pull commonworkflowlanguage/nodejs-engine

Dockstore.cwl.txt

[prismofeverything]

Now the workaround is docker pull node:slim

[tetron]

I think I know what the problem is. cwltool sets a timeout to terminate the expression so that it can't run too long. When doing docker run node:slim it exceeds the timeout the first time because it is downloading the image. The correct fix is for cwltool to run docker pull separately before docker run as suggested by @prismofeverything

[tetron]

Now pulls the image before calling docker run.