hamming: reduce number of test cases

[ErikSchierboom]

If one examines the canonical data of the hamming exercise, one finds that it has many different test cases. However, I would argue that many test cases don't add any value and go against our unwritten rule that new tests cases should require changes to the code to make them pass. In other words, at some point, the implementation will pass the first number of tests, and then most likely also automatically pass all the subsequent tests (except for the error handling ones).

IMHO the fix to this is to remove (most of) the unneeded test cases. What exactly the test cases are that should be removed is up for discussion.

[emcoding]

I agree that there're too many test cases.

I also think that the order of the cases matter. I'd like the tests to handle the happy path first, before handling the edge cases.

I'll take a look at the Ruby implementation to see what cases can be removed from a Ruby point of view. Or rather: which cases should stay.

[bitfield]

There's also the point that specific test cases can catch common mistakes. There may be 50 different common mistakes for a given exercise, of which any individual student might only make a few. Nonetheless, the cases are valuable for the population of students as a whole. (I'm not saying this is the case for Hamming necessarily; just a general observation.)

[ErikSchierboom]

I'm not saying this is the case for Hamming necessarily; just a general observation.

Well, I'm saying this is not the case for Hamming, hence the issue :)

[bitfield]

Quite. I should have made it clearer that I wasn’t disagreeing with you, just pointing out that there is sometimes value in extra test cases beyond just ‘valid output’.

[rpottsoh]

Just a quick, knee jerk, thought. If we feel there are too many [tests] then we should have some idea about which ones are unnecessary. Is this just a feeling that there are too many or is there some evidence that should be added to this discussion? Should we just pick a random case and determine its merits then move on to the next case?

I am not for or against making any changes. I am just trying to understand, better, the basis for why we think there are too many tests. I agree that unneeded cases should be removed.

[ErikSchierboom]

I am just trying to understand, better, the basis for why we think there are too many tests.

Quoting from the opening post:

[...] I would argue that many test cases don't add any value and go against our unwritten rule that new tests cases should require changes to the code to make them pass. In other words, at some point, the implementation will pass the first number of tests, and then most likely also automatically pass all the subsequent tests (except for the error handling ones).

Let's list all the non-error test cases:

"description": "empty strands",
"strand1": "",
"strand2": ""

"description": "identical strands",
"strand1": "A",
"strand2": "A"

"description": "long identical strands",
"strand1": "GGACTGA",
"strand2": "GGACTGA"

"description": "complete distance in single nucleotide strands",
"strand1": "A",
"strand2": "G"

"description": "complete distance in small strands",
"strand1": "AG",
"strand2": "CT"

"description": "small distance in small strands",
"strand1": "AT",
"strand2": "CT"

"description": "small distance",
"strand1": "GGACG",
"strand2": "GGTCG"

"description": "small distance in long strands",
"strand1": "ACCAGGG",
"strand2": "ACTATGG"

"description": "non-unique character in first strand",
"strand1": "AAG",
"strand2": "AAA"

"description": "non-unique character in second strand",
"strand1": "AAA",
"strand2": "AAG"

"description": "same nucleotides in different positions",
"strand1": "TAG",
"strand2": "GAT"

"description": "large distance",
"strand1": "GATACA",
"strand2": "GCATAA"

"description": "large distance in off-by-one strand",
"strand1": "GGACGGATTCTG",
"strand2": "AGGACGGATTCT"

IMHO, I feel that at least half, and probably even more of the test cases can be safely dropped. Consider the three test cases for a small distance:

"description": "small distance in small strands",
"strand1": "AT",
"strand2": "CT"

"description": "small distance",
"strand1": "GGACG",
"strand2": "GGTCG"

"description": "small distance in long strands",
"strand1": "ACCAGGG",
"strand2": "ACTATGG"

Couldn't these three be replaced with just the second or third test case? Furthermore, is having separate test cases for which the difference is 1 really useful when there is also a test case for a large distance? To me, the distance being 1 or 9 most likely does not mean that the user will have to change the implementation. This makes these additional tests redundant.

If I were to suggest the test cases that I find useful, I would have test cases for:

• Two empty strings
• Two single letter strings that don't match
• Two single letter strings that match
• Two longer letter strings that don't match
• Two longer letter strings that match

This means dropping the arbitrary small/large distance distinction. There is just empty strings, single letter strings and longer strings. I feel that these cases have as much chance to guide the student to a working solution as the current test cases. Having fewer test cases also has the benefit that a user is more likely to "get it",' to see what the solution is actually about.

In addition to these inputs, I would keep the error cases.

[rpottsoh]

Thanks @ErikSchierboom this is an excellent rundown and is clear to me now.

[ErikSchierboom]

My pleasure! I'll start working on a PR sometime next week. Thanks for the review.