make coverage.html; make coverage-report fails

[manu-chroma]

make coverage.html 
make coverage-report fails 

these commands fail due to the following error:

Finished processing dependencies for cwltool==1.0.20170309164828
export COVERAGE_PROCESS_START=/home/manu/github/cwltool/.coveragerc; \
       cd ; ./run_test.sh RUNNER=cwltool
/bin/sh: 2: ./run_test.sh: not found
Makefile:110: recipe for target '.coverage' failed
make: *** [.coverage] Error 127

related line of code in makefile: https://github.com/common-workflow-language/cwltool/blob/master/Makefile#L111

[manu-chroma]

I'm interested in knowing if there is any way to assess the test coverage of the current codebase.

[manu-chroma]

@mr-c thoughts ?

[mr-c]

@manu-chroma run-test.sh is from https://github.com/common-workflow-language/common-workflow-language/ -- the Makefile should be clarified -- sorry!

[manu-chroma]

@mr-c thanks. I will try this. 👍

Last night only did I realize that we can run conformance tests locally. That they are already present in cwltool/cwltool/schemas directory.

I believe that this is something that should be briefly mentioned in the main README. I was under the impression that they were only setup on the jenkins server. I'd be happy to make a PR for the same.

[mr-c]

You've touched on an organizational problem we have, I've started to think about it out loud at common-workflow-language/common-workflow-language#413

In summary, please don't use the conformance tests inside cwltool/cwltool/schemas as they are out of date :-)

[mr-c]

Yes, we should explain how to run the conformance tests in the README -- PR gladly accepted

[manu-chroma]

In summary, please don't use the conformance tests inside cwltool/cwltool/schemas as they are out of date :-)

@mr-c thanks for pointing this out. I was using them yesterday while debugging my PR: #337

Yes, we should explain how to run the conformance tests in the README -- PR gladly accepted

I'm thinking of adding short description on running tox locally. And adding essential contents from https://github.com/common-workflow-language/cwltool/blob/master/cwltool/schemas/CONFORMANCE_TESTS.md

[manu-chroma]

I was able to get combined code coverage from unit tests and conformance tests. Here it is. Conformance Tests cover roughly 30% of the codebase which is a subset of the codebase covered by unit tests. So combining the code coverage did not increase the coverage by more than few lines of code.

[mr-c]

• make a simple shell script: coverage run cwltool $@ and pass that path in as the RUNNER to gather test coverage from the conformance tests

[manu-chroma]

@mr-c little-confused here. I've created coverage.sh as: coverage run cwltool $@

How to integrate it here: https://github.com/common-workflow-language/cwltool/blob/master/Makefile#L117
I forgot what trick you told to make the RUNNER resolve correctly.

[mr-c]

@manu-chroma Did you try RUNNER=path/to/coverage.sh ?

[manu-chroma]

Yes. It's giving error in run_test.sh:

Bottom of the error message:

export COVERAGE_PROCESS_START=/home/manu/github/cwltool/.coveragerc; \
       cd common-workflow-language; ./run_test.sh RUNNER="/home/manu/github/cwltool/coverage.sh"
run_test.sh: Run common workflow tool description language conformance tests.

Syntax:
        run_test.sh [RUNNER=/path/to/cwl-runner] [DRAFT=cwl-draft-version]

Options:
  -nT   Run a specific test.
  -l    List tests

runner '/home/manu/github/cwltool/coverage.sh' not found
Makefile:116: recipe for target '.coverage' failed
make: *** [.coverage] Error 1

Going into this loop:

if ! runner="$(which $RUNNER)" ; then
    echo >&2 "$helpmessage"
    echo >&2
    echo >&2 "runner '$RUNNER' not found"
    exit 1
fi

[mr-c]

Ah, it expects it to be on the PATH. Then try

export COVERAGE_PROCESS_START=/home/manu/github/cwltool/.coveragerc; export PATH=$PATH:$PWD; \
       cd common-workflow-language; ./run_test.sh RUNNER=coverage.sh