Test for the example in CI

[tom-tan]

It would be nice if there is a CI tester (such as Travis CI) for the examples in the user guide.
Such a scheme can keep the examples runnable for each commit.

Here is the list to solve this issue:

• [cwltest] cwltest/#38 add support for "known failure" tests
• [user_guide] make the examples in each section independent
  • Categorize the examples for each section #43 Categorize the examples for each section
  • Add files for the examles in user guide #51 Add files for the examles in user guide
• [user_guide] add conformance test for the examples
  • Add conformance test for section 4 using cwltest #52 Add conformance test for section 4 using cwltest
  • Add tests for the examples #61 Add tests for the examples
    • a conformance test for section 11 (1st example)
    • conformance tests for section 2, 3
  • Add conformance tests for section 7, 9, 11 (2nd and 3rd examples), 12, 13 and 14
    • Fix an example of Section7 not to use side effect #92 for section 7
    • Fix the example for section 09 not to use side effect #94 for section 9
    • Change the examples in Section 11 not to use side-effects #98 for section 11
    • Change an example in Section 12 not to use side-effect #99 for section 12
    • Change an example in section 13 not to use side-effect #100 for section 13
    • Change an example in Section 14 not to use side-effect #101 for section 14
  • Add conformance tests for section 19 and 20
    • Add test for an example in Section 19 #102 for section 19
    • add PR for section 20 (now 19: software requirements)
    • add PR for section 22 (scatter)
• [user_guide] add files for CI for the examples
  • Add .travis.yml to test the examples #53 Add .travis.yml to test the examples
  • Enable conformance test with Travis CI #55 Enable conformance test with Travis CI

[tobyhodges]

This would be great! Perhaps I can have a go at configuring such a thing...

[tom-tan]

How about using cwltest for this purpose?
It makes easier to maintain by using the existing framework.

To use cwltest for testing user guide, there are several issues to be solved:

• In user guide, there are several examples that are expected to be failed (e.g. record.cwl and record-job1.yml in Section 11).
  It would be nice if cwltest can handle such cases.
• Some examples just output the string as a side effect and return no results (e.g. record.cwl and record-job2.yml in Section 11). Also, some examples depend on the results of other examples in previous sections (e.g. tar.cwl in Section 4).
  It would be better to fix the examples to be testable, that is, fix them not to use side effects and to be independent each other.

If it is OK for this plan, I will make issues and pull requests for it later.

[tobyhodges]

this sounds like a pretty good plan to me, though I do think that it would be good to keep examples where errors are expected. These are often good for people searching the web for an error message, and/or to demonstrate certain features e.g. dependent arguments/options.

what do you think, @mr-c?

[mr-c]

Yes, we need to be able to have "known failure" testing, so a PR to adding that to cwltest would be very welcome!

[tom-tan]

Thank you for the reply.

Here is a todo list for this issue.

• [cwltest] add support for "known failure" tests
• [user_guide] make the examples in each section independent
  • Categorize the examples for each section #43 Categorize the examples for each section
  • Add files for the examles in user guide #51 Add files for the examles in user guide
• [user_guide] make the examples not to use side effects
• [user_guide] add conformance test for the examples
  • Add conformance test for section 4 using cwltest #52 Add conformance test for section 4 using cwltest
• [user_guide] add files for CI for the examples

I will make PRs for them.

edited: I moved the above list to the first message.