Validate the test data version as part of a travis build

[behrtam]

Thanks to the work of @rbasso (see exercism/problem-specifications#602, exercism/problem-specifications#625, exercism/problem-specifications#673, etc) and many more, we now not only have a schema but also a test data version.

In exercism/problem-specifications#524 @robkeim started to think about how to propagate changes from x-common to the tracks. In this issue @ErikSchierboom wrote:

Yes, sending an update each time something changes in x-common can be tiresome, but the alternative (manually keeping track of x-common) is not a whole lot better.

So, I thought about a way that doesn't generate a lot of mails, issues or notifications, but makes sure that at least every time someone actually is working on a test file it is up to date.

I started to work on a script (check-testversion.py) for the python track that checks the versions for all exercises that are part of a travis build against the newest version in the x-common repo, a bit like a linter or stylechecker. It still needs a lot of cleanup and testing, but it at least proves that the basic idea is working.

The basic concept is ...

1. get a list of changed files git diff --name-only $TRAVIS_COMMIT_RANGE
2. get the newest version from canonical-data.json through https://api.github.com/repos/...
3. get the actual version from a comment in the test file # test @ version: 1.0.0
   (maybe it should be excluded into a separate JSON file as part of the exercise directory)
4. compare the versions and output some infos

As a test I changed the anagram version back from 1.0.1 to 1.0.0 in this commit: behrtam/python@3dc7acd
Which resulted in a failed build as expected: https://travis-ci.org/behrtam/xpython/jobs/219306945#L163

I don't know if this will finally be integrated into the python track or if this is a good idea at all, but I thought maybe some parts of this could also be interesting for other tracks.

[kytrinyx]

I like this idea a lot. I'd love to hear if any other tracks try it out and what you think.

[petertseng]

Nice. I have https://github.com/petertseng/x-common/tree/up-to-date going but haven't gone to the length of putting it in the CI for the respective tracks yet (probably shouldn't, since why would Ruby code go in non-Ruby tracks?)

That gives colour coded output of whether things are up to date, like (sorry, can't show the colour coding on Github markdown)

                  bracket-push: 1.0.0 -> 1.1.0
                       hamming: 1.0.0 == 1.0.0
                       anagram: 1.0.1 == 1.0.1
        largest-series-product: 1.0.0 == 1.0.0
                         sieve: 1.0.0 == 1.0.0
                         react: 1.0.0 == 1.0.0
                          leap: 1.0.0 == 1.0.0

[hilary]

xruby is working towards something similar. We now have the test data version in our generated test files:

• test template
• generated tests

[stevejb71]

xocaml is also doing this for generated tests, with a comment at the top of each test.ml file:

(* Test/exercise version: "1.0.0" *)

[petertseng]

I'm trying to figure out the best cadence for running a script that ensures exercises are up to date. I thought of a few choices:

• The maintainers simply have to remember to run the script on a manual but regular basis. This of course adds no complexity to the CI. The obvious disadvantage is the human element - what if someone forgets?
• CI checks changed files only. This leaves open the possibility that a file will go unchanged and therefore unchecked for a long time. How to deal with this? Combine it with the above strategy?
• CI checks all files. A big disadvantage here is that a previously-passing build may turn into failing even though nothing has changed in the repo being checked (only x-common).

I would be interested to hear any of y'all's experiences on that.

[stevejb71]

I would not want to enable the 3rd option, it would take the focus off the PR I'm working on.

Would it be possible for a central script to raise /update a single issue in each stream repo for version mismatches?

[robkeim]

@stevejb71 we have a script that can do that and we discussed enabling something like that here, although there was concern about the additional burden for track maintainers.

[stevejb71]

Can it be done on a per track basis? I'd be fine with it for the ocaml track.

[rpottsoh]

I wouldn't mind issues being created / updated in the Delphi track as long as the issue pertained to an exercise my track currently offers. Tracks should be able to opt out of the feature.

[cmccandless]

If exercism/meta#97 comes to fruition, I would prefer that solution to CI checks. While both could be done together, to me using both might feel redundant.