leap: Test case is not suitable to get desired solution.

[Stargator]

The canonical-data.json for leap only offers four cases.

Because I am mentoring on a track that implemented this, I've realized how easy it is for someone to meet the test criteria without implementing the desired solution.

Let me elaborate on how they can pass the test suite first and then I'll detail how that doesn't fit the desired solution.

The most basic solutions to pass the tests are:
First Solution (fixed a typo):

bool leapYear(int year) {
  if (year == 2015 || year == 2100 || year == 300) {
    return false;
  }
  return true;
}

Second Solution:

bool leapYear(int year) {
  if (year == 1996 || year == 2000) {
    return true;
  }
  return false;
}

The desired solution would have the algorithm:

• Check whether the year is evenly divisible by 4 then return true
• Unless the year is evenly divisible by 100
• Except if the year is also evenly divisible by 400.

The only way I see to resolve this, is just to brute force add a lot of different cases with different years.

Would that be worth it to encourage folks to actually produce a solution instead of solving the test suite?

[m-dango]

I personally would just not accept the solution and inform the submitter that there's more to an implementation than simply passing the tests. I have doubts that it would do much to encourage solving the problem over passing the test suite for people inclined to do so.

A while back I saw a submission to roman-numerals where someone just put in a dictionary, with the key as the input and the value as the expected output.

[Stargator]

@mienaikage I understand where you are getting at, but exercism also tries to support, if not encourage, people trying to solve the exercises to follow a Test Driven Development cycle.

Solve for the first exercise, then the next, refactor, solve for the next, refactor, and so on.

So I can easily see how a student could find themselves implementing such a basic solution if the tests are not "forcing" the algorithm to grow in a way that is desired.

[rpottsoh]

I don't see how either example solution would pass all of the tests. If I saw a solution like this I would ask the student to consider how their solution might cope with.... and suggest a "year" that would obviously break their solution. Encourage them to check the divisibility of the year.... Each case is pretty blatant about what to check.

[petertseng]

First Solution:

This always return true, but you meant false if 2015 or 2100, so your intent is clear even if the code needs a little fixing (I would suggest to fix it so that the intent is made clearer)

add a lot of different cases with different years.

On the surface you might think this goes against the principles in https://github.com/exercism/docs/blob/master/language-tracks/exercises/anatomy/test-suites.md where a good test case "introduces a single new requirement" and "does not duplicate the expectations of another test" , but I guess if the requirement is phrased as "doesn't look for a specific year", it could make sense.

How many is enough?

Is the number in https://github.com/exercism/problem-specifications/pull/50/files#diff-4d269b78f4f82de26d4025b6c6862148 enough?

They were removed in #463; while it may have been a good idea then, we may have new information that makes a different path a good idea now.

This is going to be true for almost any exercise; it seems that it will always be possible to submit a lookup table as an answer. There is of course the possibility of making a lookup table take too much effort compared to a more algorithmic solution.

exercism also tries to support, if not encourage, people trying to solve the exercises to follow a Test Driven Development cycle.

I see that this sentence is corroborated by https://github.com/exercism/docs/blob/master/language-tracks/exercises/anatomy/test-suites.md which has a sentence starting "We simulate Test-Driven Development (TDD) by [...]"

I can say that I prefer to look at and run all the tests at once since I like the wider view, but I can't speak to whether this style should be recommended for others.

[petertseng]

Oh, I'm sorry, I just realised my comment really isn't related to leap. Sure, it may discuss general principles that may be applied to see what should be done about leap, but the fact remains that it isn't about leap. If you wish to kick my comment to a different issue, I'm happy to have it be kicked. I can't say I'm particularly interested about discussing any one exercise in particular, which is why my comment was written the way it is.

[Stargator]

I don't see how either example solution would pass all of the tests

This always return true, but you meant false if 2015 or 2100, so your intent is clear even if the code needs a little fixing (I would suggest to fix it so that the intent is made clearer)

I fixed the first solution, I had it return true in both cases. That's been fixed.

[macta]

I’m new to these discussions, but wouldn’t the idea be to ask the student to write the next test that would cause them to refactor their solution? That’s the TDD game you play when pairing with someone ... which is the inverse of the other scenario where someone tries to overthink the problem. It can always be a learning exercise. I also think that exercism is about thinking, and playing with ideas and languages. That’s why you sign up. If you want to game it - then you ultimately hurt yourself. Tim

…

Sent from my iPhone

On 1 Aug 2018, at 22:38, Stargator ***@***.***> wrote: I don't see how either example solution would pass all of the tests I fixed the first solution, I had it return true in both cases. That's been fixed. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[Stargator]

@macta So ask the student to develop a test that would cause them to break out of their "game".

Interesting thought, but part of why I opened this issue was to see if anyone saw value in adding more test cases to the spec. Sure the track could take whatever the student did or add more tests on top of the basic. But then I wouldn't be engaging in the wider Exercism support group and see if there is actual value in it for this exercise.

This exercise is pretty basic. To make it really complicated we could add years from BCE and then label the AC years as such. That would make the input a string instead of an integer and the numbers would still have to deal with the game the student was playing.

[Stargator]

I'm willing to close this, since this really touches on a larger topic of how thorough the tests should be and whether the test should be enough to prevent the student from just hardcoding the answers.

And because that's a larger question that affects all Exercism exercises, maybe this discussion is warranted elsewhere?

[rpottsoh]

I think the burden is on the mentors and the students in this scenario. I don't think changing the testdata will prevent a student from coming up with a solution that satisfies the test but misses the mark on the desired intent o the exercise.

Applying measures to the testdata to prevent the student from being lazy are not easily accomplished without exposing the desired solution to the student in the test suite.

I agree that this issue can probably be closed. This is an interesting issue, worthy of discussion in another area. I think the general rule of thumb simply is for the mentor to spot what is happening and encourage the student to try another approach. If the student is unwilling / unable to apply the correct algorithm the burden is then on the mentor to decide to approve (or not) the student's work. The mentor may want to confer with others at this point to decide to approve (or not) the students work. The student will likely not last for long anyway if they are really not into the spirit of Exercism.

[Stargator]

@rpottsoh Agreed, thanks. Closing the issue.