Description
(status: I am currently working on learning how to fix, and then fixing, this bug :) Pull request submitted!)
The following tests are supposed to be checking that they get an exception when they ask for the score when a game is incomplete for various reasons:
test_an_unstarted_game_cannot_be_scoredtest_an_incomplete_game_cannot_be_scoredtest_bonus_rolls_for_a_strike_in_the_last_frame_must_be_rolled_before_score_can_be_calculatedtest_both_bonus_rolls_for_a_strike_in_the_last_frame_must_be_rolled_before_score_can_be_calculatedtest_bonus_roll_for_a_spare_in_the_last_frame_must_be_rolled_before_score_can_be_calculated
However, all five of these tests make a call to game.roll() without passing any arguments. game.roll() is supposed to be passed a number of pins on a roll; when that argument is missing, it raises a TypeError. Those five tests, which were expecting an exception to be raised, therefore pass - but the exception that was raised was nothing to do with the game being complete or incomplete (which the tests are supposed to be checking for); the exception was actually just a consequence of the malformed call to the roll method.
Comparing the Python test file with the Ruby test file for the same exercise shows that the Ruby ones call the game.score method, not the game.roll method, for those five tests, which indicates that the problem is actually in the Python test file generator (JinJa2 template), specifically this block of code (lines 7-10:
with self.assertRaisesWithMessage(Exception):
game.roll({{ input["roll"] }})
{% else -%}
In error_case type tests, there is no consideration of the property field from the canonical-data.json file; so the tests that expect an exception to be raised are always generated with game.roll() with no arguments, and thus they always do generate exceptions, and thus they always pass, regardless of what the user has coded.
Contrast with the corresponding code in the Ruby test file generator, lines 31-35, which correctly use the property field to decide which function call should be used for these tests.