Versioning guidelines

[NobbZ]

Today 3 PRs have been made (perhaps more that slipped my RADAR), which all bumped MAJOR version, because of trivial changes.

In all 3 PRs voices rose that wanted a less drastic version bump:

• queen-attack: represent positions as objects #935 and robot-simulator: encode position as object instead of string #936: It was requested that only PATCH was bumped, since the actual testdata didn't change, only its source representation. No track had to change its language specific representation of the testdata to actually comply to the new testdata.
• crypto-square: Improve test data #937: It was requested that only MINOR was bumped, because the tests stayed the same, but were only slightly re-orderd, a single one added and also de-nested.

But the current versioning and bumping guidelines tell us this:

• MAJOR changes should be expected to break even well-behaved test generators.
• MINOR changes would never break well-designed test generators, because the test-generation logic remains exactly the same.
• PATCH changes would never break well-designed test generators, because the test data remains exactly the same

[enumeration mine, detailed descriptions omitted for now]

After reading this TL;DRs of the various kinds of version bumps, some question arise.

• What is a “well-behaved” testgenator?
• What changes except for renaming, adding, removing or reordering tests, will not break a generator?

Even worse, on a further read of the more thorough description of each bump I found this:

[bump PATCH when] Regrouping/"Renesting" test cases without changing test case ordering.

But how is changing the nesting, not changing the logic of the testgenerator? It can't find the tests anymore where it expects them.

Also I do feel, that adding and removing testcases, may end in a completely different way to solve the exercise, which makes the whole testsuite incompatible with its previous version, so it should be a MAJOR bump here, instead of a MINOR as in the document:

[bump MINOR when] Adding or deleting test cases.

Nearly the same reasoning applies in my eyes for the other two cases where a MINOR were sufficient, but I'd do a MAJOR because it changes the way to solve the exercise drastically:

• Changing the test cases inputs and/or outputs.
• Changing the test cases ordering.

---

When we introduced the versioning scheme, I've understood it as versioning the exercise, today I learnt it's meant to version the generator.

I do not think this is the way to go, but I will not try to change that anymore, I think I'm just to late to the party to change this, but what I can try, is to simplify the current bumping guide and remove ambiguity:

• Bump MAJOR when you do not bump MINOR
• Bump MINOR on fixing typos in comments and descriptions or you are reformatting/reordering the JSON document without changing its semantics.
• Remove PATCH from versioning scheme

Basically, this would mean, everything which could break existing solutions or examples or testgenerators gets a MAJOR bump. MINOR bumps are for semantically no-ops…

Edit: Fix typos

[petertseng]

When we introduced the versioning scheme, I've understood it as versioning the exercise, today I learn't its meant to version the generator.

That doesn't really make sense, since the generators, if they exist at all, live in individual repositories. But the goal does inform how track maintainers may treat their test generators.

I thought the goal is this: Allow track maintainers to understand whether the canonical-data has changed since it was last used, and if it has changed, the nature of the changes.

Part of it does relate to test generators, but it seems that it's useful for humans too. It's useful for even tracks that don't have test generators, right?

It looks like the intent was mirrored in #673. The original author has left this organisation, so an attempt to directly ask the original author may not necessarily be effective, but it can be tried.

I thought the intent was good.

I agree that changing nesting would be larger than patch.

I have not judged whether each other item is in the correct category given its intent.

[NobbZ]

When we introduced the versioning scheme, I've understood it as versioning the exercise, today I learn't its meant to version the generator.

That doesn't really make sense, since the generators, if they exist at all, live in individual repositories. But the goal does inform how track maintainers may treat their test generators.

Yeah, in lack of a better wording, this is what I've written yesterday, and even though I had a lot of time to think about it, I have trouble finding correct words…

So I try it the other way round…

As a user, I do expect that if I see a major bump in the testdata, I do expect my solution to break. I expect the resulting testsuite to be different than before.

But in the current version scheme, if I see a major bump, all it tells me, the change in the data broke the program that generates the testsuite, if there is such a program at all… Even worse, we have proves of changes that require a major bump, but will result in exactly the same testsuite after the generator has been repaired.

This is counter intuitive!

Especially given the fact, that currently in at least one track (Haskell[1]) the version is even user faced, I feel confused that a version change indicating a bump, does not even change the testsuie…

[1] : Originally I wanted to copy their version schema, but since v2 is on the doorstep and it will bind a submission to the testsuite of the student, I dropped that plan.

[Insti]

It's tricky because I think we're trying to track two things at once.
Should we add another version key? So we're tracking both testcases version and test metadata version?

[petertseng]

As a user, I do expect that if I see a major bump in the testdata, I do expect my solution to break. I expect the resulting testsuite to be different than before.

we have proves of changes that require a major bump, but will result in exactly the same testsuite after the generator has been repaired.

Oh, I see. This looks like a problem to me because it goes against the goal of knowing "has canonical data changed"? There are major version changes where the canonical data should not be considered to change, only its JSON representation.

Put another way, I originally wrote in #673 (comment) ...

Major: You might need to rewrite your test generator.
Minor: You will definitely not need to rewrite your test generator. You probably want to rerun your test generator though.
Patch: Rerunning the test generator is probably not necessary.

However, we may have a "Major" change that indicates you may need to rewrite your test generator... but you don't need to rerun it even after you rewrite it because the generated test suite would be the same. Oops.

It's hard to come up with a scheme that answers both the questions "do I need to rewrite and/or rerun my generators" and "did the tests change?".

One possibility is two monotonically increasing version components that increase independently from one another (the proposal of Independent Components). If canonical data is version X.Y:

• If test generator would need to be rewritten, but the generated tests will not change, next version is X.(Y+1)
• If the generated tests will change but the generator does not need to be rewritten, next version is (X+1).Y
• If both statements are true, next version is (X+1).(Y+1)

Seems too confusing and I'm going to mix up the meanings of the two components all the time. I don't like the proposal of Independent Components.

So I'm going to write this proposal. Is it still useful to know whether I might change a generator, even if it less important than knowing whether the tests changed? Then the proposal of Tests-Generators-Patch: If canonical data is version X.Y.Z:

• If tests change, next version is (X+1).0.0. It is not stated whether test generator will need to be rewritten.
• If tests do not change but test generator needs to be rewritten, next version is X.(Y+1).0.
• If tests do not change and test generator does not need to be rewritten, next version is X.Y.(Z+1).

I'm re-reading Tests-Generators-Patch and I don't think it's very useful anyway because no matter what, if I see a change in problem-specifications, I'm going to want to follow those changes, and I'll find out then whether generator needs to be rewritten. It's not very interesting to encode it in the version scheme, and it imposes additional burden since it may incite questions about whether a given change would require a generator rewrite.

OK, I say I support the proposal of just Major/Minor as noted in #938 (comment) .

[HarrisonMc555]

At first I thought I supported the simple Major/Minor version as well, but we still have the issue of when the JSON representation changes in a way that would break generators but the test data will remain the same. It seems to me that we really are trying to keep track of two different things for two different audiences:

1. The writers of the tests/test generators
2. The writers of the solutions/examples

There are situations in which one could change but not the other and situations in which both could change together.

Although I know it's probably not popular or preferred, it seems like the best solution would be to have two different version numbers: one for the test cases themselves (semantics) and one for the representation (syntax).

We could possibly have two different keys.

• version key (for semantics)
• jsonVersion key (for syntax)

If version is incremented, then I may need to rewrite my solution/example and re-run my test generator. If jsonVersion is incremented, I may need to rewrite my test generator. If both are incremented, then I need to do both.

I'm basically saying the same thing as @petertseng but I'm in favor having two different keys, because they really are keeping track of two totally different things.

[petertseng]

So, ultimately I would be happy with having both version keys. I would want others to make that decision only if both keys were useful. I want to think more about whether it is useful to track the structure version.

What would I, as a track maintainer, probably want to do in the three different cases?

• Semantics change, but structure doesn't: Rerun the test generator. Easy.
• Semantics and structure change: Rerun the test generator, after it has been rewritten.
• Structure changes, semantics don't: So, maybe I would rewrite my test generator if I felt like it
  • Doesn't seem very useful, since I don't need to rerun it after I finished rewriting it, other than to verify that the rewrite worked.
  • What if then there is another structural change? If I had waited, I would only have to do one rewrite instead of two.

So, I need to rerun the generator if and only if the semantics change. Therefore, it is indeed important to let maintainers know about semantics changes.

It does not seem necessary in the same way to let maintainers know about structural changes. Here is how I decide what to do the version number only indicates "is there a semantics change?"

• Version number indicates there is one. I rerun. I might have to rewrite the generator. I will find this out when I run the generator, instead of by looking at some version number.
• Version number indicates there is none. I do not rerun.

I leave it to y'all to decide whether being able to tell ahead-of-time whether I need to rewrite a generator is worth the increased complexity of two separate version numbers. I say it is not.

---

Fun Fact: I do not currently maintain any track that uses test generators.

[HarrisonMc555]

I like what @petertseng says. This sounds good to me.

---

I also do not maintain any tracks, so maybe I shouldn't be part of this discussion 😉

[Vankog]

Funny to find this issue, because I was about to open a similar one and changed my mind midway through.

In my eyes some changes are not worth a minor, but a patch. For example, changing the ordering of cases creates semantically the same test suite. It only affects the end user's progression.

With the current definition, basically ONLY the minor version level has any relevance.
The Major will probably never change and pure Patches seem quite rare.

e.g. in #957 I had to bump the Minor versioning, just because of an ordering-bug I introduced just prior to that.

[Insti]

I leave it to y'all to decide whether being able to tell ahead-of-time whether I need to rewrite a generator is worth the increased complexity of two separate version numbers. I say it is not.

I agree that it is not worth tracking separate version numbers and the version key should only track semantic changes that will cause changes to the generated test cases.

If you really care, structural or other changes that don't affect the test output can be detected via the git logs / commit hashes.

---

I do maintain a track that uses test generators.

And I don't even look at the version numbers, I just periodically regenerate the tests and look at the diffs to see what has changed. If any generators break I'll fix them. This is usually a simple task.

[petertseng]

By suggesting an extreme option, I hope to reveal what we want by listening to reasons why we reject this extreme option.

Let's remove versioning completely. We'll rely on git sha exclusively.

I think my reason for rejecting this is that sometimes the file changes but the tests don't change, and I don't want my up-to-date scripts to tell me merely that the file changed.

I think at this point all I care about is whether the tests will change (regardless of whether I need to rewrite a generator). In fact, I think that means I only need a single-component version, and that version changes iff tests change. I wouldn't oppose a version with more components, but they may mean more opportunities to forget bumping components.

[Insti]

Let's remove versioning completely. We'll rely on git sha exclusively.

I like this suggestion.

sometimes the file changes but the tests don't change, and I don't want my up-to-date scripts to tell me merely that the file changed.

Why cannot your up-to-date scripts regenerate the tests and only tell you if they change.

[petertseng]

sometimes the file changes but the tests don't change, and I don't want my up-to-date scripts to tell me merely that the file changed.

Why cannot your up-to-date scripts regenerate the tests and only tell you if they change.

Well, I was kind of hoping that the scripts can be run on any repository, and thus they have no language-specific generation functionality.

I suppose I could do some minimal parsing and generate tests in some arbitrary target language (doesn't even have to be a real language!) and thereby determine whether two JSON files represent the same set of tests. I'll chew on the idea and think about whether I find it useful. It's nonzero work, but some of the work has been done already (verify).