Ideas for further improvement

[Timendus]

• Add a specific XO-CHIP test (more memory, planes, scrolling, save and load (flags), long load i (with proper skip))
• Generate screenshots of all possible permutations of opcodes not being implemented, so we get a "what's wrong when I see this" page somewhere
• Add tests to keypad test to check if key is properly &'ed with 0xF
• Add tests to see if wrapping and clipping work properly even if drawing 16x16 sprites or four colour sprites. Seems like these are often made mistakes.

[Timendus]

Things resolved in upcoming version 4:

• The test suite has been taken apart into stand alone binaries for each test. The menu is only used for the quirks and the keypad test, and can still be overridden by setting RAM[0x1FF] to a magic value. All tests now have the test suite version number shown in them somewhere.
• The flags test is more robust and now also checks for cases where vX is vF, in addition to the old checks where vY is vF. It also now checks if i += vF works like it should.
• The Corax89 test is now the Corax+ test. It has four additional tests that the original doesn't have, one of which checks to see if the registers are 8 bits wide. The others are regular opcode tests that were just missing
• The quirks test looks better and gives better feedback when things go wrong

[jon-axon]

Something else that might be nice to add (now that program memory won't be a problem going forwards with the stand alone binaries) is a sound test. I don't think the test suite will be able to automatically check and report on the implementation in the same way it does with flags, quirks etc, but it could be nice to have a screen on which a message is displayed stating an audio pattern (e.g. something like a 1 second beep following by 1 second of silence then 5 beeps of half a second each separated by half a second of silence) and then the test suite plays the corresponding audio. The user can listen and easily check that the audio they hear being played by the interpreter matches the stated pattern in the test suite.

I would certainly have found this useful recently for verifying my own interpreter audio playback. Instead I had to randomly pick existing ROMs then run them side-by-side in my emulator and other established emulators to ensure the audio matched; having a specific test would make this easier.

[Timendus]

For the subtraction opcodes, the test currently doesn't check how the interpreter sets the flag if the subtracted values are equal, which is apparently an annoying edge case for some people. Could be nice to add a test for that.

[gulrak]

For the subtraction opcodes, the test currently doesn't check how the interpreter sets the flag if the subtracted values are equal, which is apparently an annoying edge case for some people. Could be nice to add a test for that.

Just a small note: This edge case makes the quirks test report "LOW" for the display wait quirk for an emulator that correctly implements the quirk, making the devs hunt in the wrong direction, so if other tests depend on the edge case it should be tested in the flags test, I think.

[Timendus]

Things resolved in upcoming version 4.1:

• Add a test for the scrolling instructions
• Add a beeping test
• Check subtraction flag for equal values
• Solve SUPER-CHIP compatibility issues of the quirks test

[Timendus]

This was resolved by #22 (but not yet released)

• Test the BCD opcode FX33 with a smaller value, to catch issues where the numbers are stored in the wrong bytes

[Timendus]

Things resolved in version 4.2:

• The Corax+ test now properly tests both calling and returning, as well as BCD with values that result in one or two digits, thanks to @GamingMadster (Improve Corax+ #22)
• The README has once again been improved, thanks to @greg-kennedy (Keypad test: More info on NOT HALTING #20) and feedback from @algometric (No-op passing several corax tests #19)
• The quirks test now tests the memory quirk for reading and writing, not just writing. Thanks to @gfcwfzkm for reporting this issue (Quirks test Memory is only testing FX55, not FX65 #21)

[eshom]

CHIP-8 beginner here! First of all thank you for your test suite, it has been an invaluable source so far in my journey!

Now to the suggestion:
Is there a test for CXNN?
It doesn't make much sense to test the rng part (unless you really care about statistical distributions), but you CAN test the mask was implemented correctly and not forgotten.

Not sure what kind of problems you might encounter if you forget the mask, but would be nice to check for it, no?

[gulrak]

CHIP-8 beginner here! First of all thank you for your test suite, it has been an invaluable source so far in my journey!

Now to the suggestion: Is there a test for CXNN? It doesn't make much sense to test the rng part (unless you really care about statistical distributions), but you CAN test the mask was implemented correctly and not forgotten.

Not sure what kind of problems you might encounter if you forget the mask, but would be nice to check for it, no?

Well, you have to use the (P)RNG part to test for the mask, so the test needs to loop to make sure that enough bits come together to check for the effect of the mask.

I do this in my unit tests, where I execute Cxnn with fixed masks a bunch of times and AND/OR together the returns and check that after enough iterations the AND result is equal zero (to check there are no stuck bits in the RNG) and the OR result is equal to the mask, showing that each mask bit was set at least in one result. Without a "working" RNG I see no way to validate the working of the mask from the perspective of the CHIP-8 program.

[gulrak]

Btw. even the not really good random generator in the original COSMAC VIP manages to succeed in that test after 16 iterations, so it should not need too many of them, so a test could run in a loop and exit if the AND result (starting at 0xFF) and the OR result (starting as 0x00) have reached 0/mask or a max tries counter ran down (255 should be easily enough and would not run that long), making the reach of the end of the counting a fail for Cxnn and the early due to both values being correct a success.

Haven't done this as a CHIP-8 program yet, to validate the quality of the HP-48SX implementations yet though.

[eshom]

Btw. even the not really good random generator in the original COSMAC VIP manages to succeed in that test after 16 iterations, so it should not need too many of them, so a test could run in a loop and exit if the AND result (starting at 0xFF) and the OR result (starting as 0x00) have reached 0/mask or a max tries counter ran down (255 should be easily enough and would not run that long), making the reach of the end of the counting a fail for Cxnn and the early due to both values being correct a success.

Haven't done this as a CHIP-8 program yet, to validate the quality of the HP-48SX implementations yet though.

Thank you! I have improved my unit test with your suggestion. Still would be fun to have this kind of test as part of the suite. I might try making this after I'm done with everything else.

[Timendus]

That does sound like it could be a valuable addition to the test suite. And Gulrak's solution is also how I would approach this problem.

I would like to add that it would need to run this test several times, cycling through different masks, to make sure the mask actually works like a mask. We also have to be very explicit in the manual that this test can in some cases produce false negative results.