Document how to run the cwltool tests

[peterjc]

I tried this, based on the install from source instructions and typical Python setup.py usage:

$ git clone https://github.com/common-workflow-language/cwltool.git
...
$ cd cwltool
$ python2.7 setup.py build
...
$ python2.7 setup.py test
running test
running egg_info
writing requirements to cwltool.egg-info/requires.txt
writing cwltool.egg-info/PKG-INFO
writing top-level names to cwltool.egg-info/top_level.txt
writing dependency_links to cwltool.egg-info/dependency_links.txt
writing entry points to cwltool.egg-info/entry_points.txt
reading manifest file 'cwltool.egg-info/SOURCES.txt'
writing manifest file 'cwltool.egg-info/SOURCES.txt'
running build_ext
test_factory (tests.test_examples.TestFactory) ... [job 139890816825808] /tmp/tmpqoAycd$ echo foo > /tmp/tmpqoAycd/out.txt
'echo' not found
[job 139890816825808] completed permanentFail
Final process status is permanentFail
ERROR
test_params (tests.test_examples.TestParamMatching) ... ok

======================================================================
ERROR: test_factory (tests.test_examples.TestFactory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/shared/users/xxx/repositories/cwltool/tests/test_examples.py", line 107, in test_factory
    self.assertEqual(echo(inp="foo"), {"out": "foo\n"})
  File "/mnt/shared/users/xxx/repositories/cwltool/cwltool/factory.py", line 11, in __call__
    return self.factory.executor(self.t, kwargs, os.getcwd(), None, **self.factory.execkwargs)
  File "/mnt/shared/users/xxx/repositories/cwltool/cwltool/main.py", line 171, in single_job_executor
    raise workflow.WorkflowException("Process status is %s" % (final_status))
WorkflowException: Process status is ['permanentFail']

----------------------------------------------------------------------
Ran 2 tests in 3.838s

FAILED (errors=1)

As an aside, the apparent failure to run the echo command is odd:

$ echo foo
foo
$ more /tmp/tmpQWsOo2/out.txt
foo

I am assuming this failure is down to the setup, rather than a bug in the test suite?

See also #37 for using the tests within TravisCI.

[mr-c]

Thanks @peterjc for reminding us.

The tests now run the normal way you tried above. Most testing is focused on improving the CWL conformance tests

[peterjc]

Confirming running tests this way now works for me,

$ git clone https://github.com/common-workflow-language/cwltool.git
...
$ cd cwltool
$ python2.7 setup.py build
...
$ python setup.py test
running test
running egg_info
writing requirements to cwltool.egg-info/requires.txt
writing cwltool.egg-info/PKG-INFO
writing top-level names to cwltool.egg-info/top_level.txt
writing dependency_links to cwltool.egg-info/dependency_links.txt
writing entry points to cwltool.egg-info/entry_points.txt
reading manifest file 'cwltool.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'cwltool.egg-info/SOURCES.txt'
running build_ext
test_factory (tests.test_examples.TestFactory) ... [job echo.cwl] /var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmpojwj3G$ echo \
    foo > /var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmpojwj3G/out.txt
Final process status is success
ok
test_params (tests.test_examples.TestParamMatching) ... ok
test_scandeps (tests.test_examples.TestScanDeps) ... [
    {
        "basename": "bar.cwl", 
        "class": "File", 
        "location": "file:///example/bar.cwl"
    }, 
    {
        "basename": "data2.txt", 
        "class": "File", 
        "location": "file:///example/data2.txt"
    }, 
    {
        "basename": "data.txt", 
        "class": "File", 
        "location": "file:///example/data.txt"
    }
]
ok
test_recordcompare (tests.test_examples.TestTypeCompare) ... ok
test_typecompare (tests.test_examples.TestTypeCompare) ... ok
test_subclass (tests.test_pathmapper.TestPathMapper) ... ok
test_bool (tests.test_toolargparse.ToolArgparse) ... setup.py 1.0.20160720090603
usage: /var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmp9Wz0Hp
       [-h] --bdg [job_order]

positional arguments:
  job_order   Job input json file

optional arguments:
  -h, --help  show this help message and exit
  --bdg
ok
test_help (tests.test_toolargparse.ToolArgparse) ... setup.py 1.0.20160720090603
Parsed job order from command line: {
    "input": {
        "class": "File", 
        "location": "file:///private/tmp/cwltool/README.rst"
    }, 
    "id": "/var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmpEJ_o9E", 
    "job_order": null
}
[job tmpEJ_o9E] initializing from file:///var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmpEJ_o9E
[job tmpEJ_o9E] {
    "input": {
        "basename": "README.rst", 
        "class": "File", 
        "location": "file:///private/tmp/cwltool/README.rst"
    }, 
    "job_order": null
}
[job tmpEJ_o9E] path mappings is {
    "file:///private/tmp/cwltool/README.rst": [
        "/private/tmp/cwltool/README.rst", 
        "/var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmp7j7xCP/stg91db475b-fecc-4430-8228-ae1356adbbdb/README.rst", 
        "File"
    ]
}
[job tmpEJ_o9E] command line bindings is [
    {
        "position": [
            -1000000, 
            0
        ], 
        "valueFrom": "cat"
    }, 
    {
        "position": [
            0, 
            "input"
        ], 
        "valueFrom": {
            "basename": "README.rst", 
            "nameroot": "README", 
            "nameext": ".rst", 
            "location": "file:///private/tmp/cwltool/README.rst", 
            "path": "/var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmp7j7xCP/stg91db475b-fecc-4430-8228-ae1356adbbdb/README.rst", 
            "dirname": "/var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmp7j7xCP/stg91db475b-fecc-4430-8228-ae1356adbbdb", 
            "class": "File"
        }
    }
]
[job tmpEJ_o9E] /var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmpHA7GGc$ cat \
    /var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmp7j7xCP/stg91db475b-fecc-4430-8228-ae1356adbbdb/README.rst > /var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmpHA7GGc/test.txt
[job tmpEJ_o9E] completed success
[job tmpEJ_o9E] {
    "output": {
        "checksum": "sha1$f566398ee8862d9c1c5ed894f7c344a9e6658167", 
        "basename": "test.txt", 
        "size": 2077, 
        "location": "file:///var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmpHA7GGc/test.txt", 
        "class": "File"
    }
}
Final process status is success
[job tmpEJ_o9E] Removing input staging directory /var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmp7j7xCP
[job tmpEJ_o9E] Removing temporary directory /var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmpXLdJNC
Moving /var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmpHA7GGc/test.txt to /private/tmp/cwltool/test.txt
Removing intermediate output directory /var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmpHA7GGc
{
    "output": {
        "checksum": "sha1$f566398ee8862d9c1c5ed894f7c344a9e6658167", 
        "basename": "test.txt", 
        "location": "file:///private/tmp/cwltool/test.txt", 
        "path": "/private/tmp/cwltool/test.txt", 
        "class": "File", 
        "size": 2077
    }
}
setup.py 1.0.20160720090603
Parsed job order from command line: {
    "input": {
        "class": "File", 
        "location": "file:///private/tmp/cwltool/README.rst"
    }, 
    "id": "/var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmpEJ_o9E", 
    "job_order": null
}
[job tmpEJ_o9E_2] initializing from file:///var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmpEJ_o9E
[job tmpEJ_o9E_2] {
    "input": {
        "basename": "README.rst", 
        "class": "File", 
        "location": "file:///private/tmp/cwltool/README.rst"
    }, 
    "job_order": null
}
[job tmpEJ_o9E_2] path mappings is {
    "file:///private/tmp/cwltool/README.rst": [
        "/private/tmp/cwltool/README.rst", 
        "/var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmphEtxSq/stgb76abc0e-df31-4ab1-b16e-c1c8ef63ff71/README.rst", 
        "File"
    ]
}
[job tmpEJ_o9E_2] command line bindings is [
    {
        "position": [
            -1000000, 
            0
        ], 
        "valueFrom": "cat"
    }, 
    {
        "position": [
            0, 
            "input"
        ], 
        "valueFrom": {
            "basename": "README.rst", 
            "nameroot": "README", 
            "nameext": ".rst", 
            "location": "file:///private/tmp/cwltool/README.rst", 
            "path": "/var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmphEtxSq/stgb76abc0e-df31-4ab1-b16e-c1c8ef63ff71/README.rst", 
            "dirname": "/var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmphEtxSq/stgb76abc0e-df31-4ab1-b16e-c1c8ef63ff71", 
            "class": "File"
        }
    }
]
[job tmpEJ_o9E_2] /var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmpJUCo31$ cat \
    /var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmphEtxSq/stgb76abc0e-df31-4ab1-b16e-c1c8ef63ff71/README.rst > /var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmpJUCo31/test.txt
[job tmpEJ_o9E_2] completed success
[job tmpEJ_o9E_2] {
    "output": {
        "checksum": "sha1$f566398ee8862d9c1c5ed894f7c344a9e6658167", 
        "basename": "test.txt", 
        "size": 2077, 
        "location": "file:///var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmpJUCo31/test.txt", 
        "class": "File"
    }
}
Final process status is success
[job tmpEJ_o9E_2] Removing input staging directory /var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmphEtxSq
[job tmpEJ_o9E_2] Removing temporary directory /var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmpGts7ST
Moving /var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmpJUCo31/test.txt to /private/tmp/cwltool/test.txt
Removing intermediate output directory /var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/tmpJUCo31
{
    "output": {
        "checksum": "sha1$f566398ee8862d9c1c5ed894f7c344a9e6658167", 
        "basename": "test.txt", 
        "location": "file:///private/tmp/cwltool/test.txt", 
        "path": "/private/tmp/cwltool/test.txt", 
        "class": "File", 
        "size": 2077
    }
}
ok

----------------------------------------------------------------------
Ran 8 tests in 1.591s

OK

Should the main README file talk about how to run the tests?