Coverage reporting

[psafont]

This adds coverage reporting and uploading to both coveralls and codecov, any of which can be easily disabled.

A common-workflow-language admin should give access to codecov or coveralls to enable uploads to them.

This PR supersedes #526

[mr-c]

Can you also look at the Makefile which has some related targets? It is good to enable local feedback too. Thanks!

[psafont]

What is the functionality that is needed in the Makefile?
Generate coverage of all the tests on the default python version and generate a report?

[mr-c]

Yes, which might already exist but should be documented in our non-existent developer guide 😋

[psafont]

So this is the result I have right now:

$ make coverage
coverage report
Name                               Stmts   Miss Branch BrPart  Cover
--------------------------------------------------------------------
cwltool/__init__.py                    2      0      0      0   100%
cwltool/__main__.py                    4      4      0      0     0%
cwltool/argparser.py                 202     75     50      2    51%
cwltool/builder.py                   206     62    138     26    62%
cwltool/command_line_tool.py         477    214    246     44    50%
cwltool/cwlrdf.py                     84     34     38     11    53%
cwltool/docker.py                    192    167    116      0     8%
cwltool/docker_id.py                  44     31      8      0    25%
cwltool/draft2tool.py                  3      3      0      0     0%
cwltool/errors.py                      4      0      0      0   100%
cwltool/executors.py                 105     46     42     11    49%
cwltool/expression.py                185     32    106     18    78%
cwltool/factory.py                    41      1      6      2    94%
cwltool/flatten.py                    19      5     10      3    72%
cwltool/job.py                       295    130    156     28    49%
cwltool/load_tool.py                 192     57    112     22    65%
cwltool/loghandler.py                  5      0      0      0   100%
cwltool/main.py                      381    180    200     54    48%
cwltool/mutation.py                   39     11      8      2    64%
cwltool/pack.py                      151    136    124      0     5%
cwltool/pathmapper.py                191     58    100     15    65%
cwltool/process.py                   569    207    346     41    57%
cwltool/resolver.py                   40     30     18      0    17%
cwltool/sandboxjs.py                 206    105     94     20    44%
cwltool/secrets.py                    41     24     38      3    33%
cwltool/singularity.py               109     88     54      0    13%
cwltool/software_requirements.py      74     53     26      0    21%
cwltool/stdfsaccess.py                51      8     14      3    77%
cwltool/update.py                    119     90     60      0    16%
cwltool/utils.py                     104     63     60      4    36%
cwltool/workflow.py                  647    165    390     54    73%
--------------------------------------------------------------------
TOTAL                               4782   2079   2560    363    51%

as a by-product xml and html reports are also generated.

[codecov-io]

Codecov Report

❗ No coverage uploaded for pull request base (master@8194b56). Click here to learn what that means.
The diff coverage is n/a.

@@            Coverage Diff            @@
##             master     #691   +/-   ##
=========================================
  Coverage          ?   62.37%           
=========================================
  Files             ?       26           
  Lines             ?     4274           
  Branches          ?     1147           
=========================================
  Hits              ?     2666           
  Misses            ?     1325           
  Partials          ?      283

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8194b56...bdd6615. Read the comment docs.

[mr-c]

can we keep the HTML report?

[psafont]

In retrospect, we can also keep the xml report

[manu-chroma]

this is awesome!