Fix exec_js_process process caching

[ThomasHickman]

The existing logic doesn't properly cache processes, leading to a new process being created for every invocation of exec_js_process (eventually leading to an error due to too many processes with many calls to execjs). This PR fixes this logic and adds tests for this.

[tetron]

@ThomasHickman

I updated arvados-cwl-runner with the latest cwltool with the jshint feature, and we are seeing some intermittent failures in our internal testing:

04:31:24 Test [55/127] 
04:31:24 
04:32:03 Test failed: /data-xvdf/home/ci/arvados-cwl-runner-with-checksum.sh --outdir=/tmp/tmpDkOqDj --quiet 'v1.0/search.cwl#main' v1.0/search-job.json
04:32:03 Test InitialWorkDirRequirement linking input files and capturing secondaryFiles
04:32:03 on input and output.
04:32:03 
04:32:03 Returned non-zero
04:32:03 2018-03-24 04:32:02 cwltool ERROR: Unhandled error, try again with --debug for more information:
04:32:03   [Errno 12] Cannot allocate memory

04:37:28 Test [61/127] 
04:37:28 
04:37:58 Test failed: /data-xvdf/home/ci/arvados-cwl-runner-with-checksum.sh --outdir=/tmp/tmpZKpRlA --quiet v1.0/params2.cwl v1.0/empty.json
04:37:58 Test parameter evaluation, with support for JS expressions
04:37:58 
04:37:58 Returned non-zero
04:37:58 2018-03-24 04:37:58 cwltool ERROR: I'm sorry, I couldn't load this CWL file, try again with --debug for more information.
04:37:58 The error was: [Errno 12] Cannot allocate memory

On a different test run:

04:43:04 Test [61/127] 
04:43:04 
04:45:53 Test failed: /home/ci/arvados-cwl-runner-with-checksum.sh --outdir=/tmp/tmp0TsfdP --quiet v1.0/params2.cwl v1.0/empty.json
04:45:53 Test parameter evaluation, with support for JS expressions
04:45:53 
04:45:53 Returned non-zero
04:45:53 2018-03-24 04:45:52 cwltool WARNING: jshint process timed out
04:45:53 2018-03-24 04:45:52 cwltool ERROR: Unhandled error, try again with --debug for more information:
04:45:53   jshint failed to run succesfully
04:45:53 returncode: -1
04:45:53 stdout: ""
04:45:53 stderr: ""

04:50:59 Test [67/127] 
04:50:59 
04:51:14 Test failed: /home/ci/arvados-cwl-runner-with-checksum.sh --outdir=/tmp/tmpP077ws --quiet v1.0/vf-concat.cwl v1.0/empty.json
04:51:14 Test that second expression in concatenated valueFrom is not ignored
04:51:14 Returned non-zero
04:51:14 2018-03-24 04:51:13 cwltool ERROR: Unhandled error, try again with --debug for more information:
04:51:14   jshint failed to run succesfully
04:51:14 returncode: 125
04:51:14 stdout: ""
04:51:14 stderr: "docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
04:51:14 See"

Do you think these are caused by the bug fixed by this PR, or does it look like another problem?

[tetron]

Also, your jshint unit tests appear to be failing with this fix.

[codecov-io]

Codecov Report

Merging #701 into master will increase coverage by 0.1%.
The diff coverage is 100%.

@@            Coverage Diff            @@
##           master     #701     +/-   ##
=========================================
+ Coverage   62.46%   62.56%   +0.1%     
=========================================
  Files          27       27             
  Lines        4422     4429      +7     
  Branches     1182     1184      +2     
=========================================
+ Hits         2762     2771      +9     
+ Misses       1366     1365      -1     
+ Partials      294      293      -1

Impacted Files	Coverage Δ
cwltool/sandboxjs.py	46.41% <100%> (+2.5%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f10d578...42e69dc. Read the comment docs.

[ThomasHickman]

@tetron

Do you think these are caused by the bug fixed by this PR, or does it look like another problem?

This could possibly be fixed by this PR, as the effects of this problem would be that a new process is generated for every JavaScript expression - which may result in the memory issues above.