Unified Sub-package Testing

[seanpmorgan]

Similar to Keras layer_test it would be nice to have a suite of tests that all addons in a subpackage are verified against.

We've seen a number of times an addon which looked correct but later failed for serialization or something along those lines since it was missed in review. Maintainers try our best on reviews, but its volunteer work and things will always get through.

Any thoughts on the complexities of implementing this? If not we can move forward with tracking the implementation of these.

[gabrieldemarmiesse]

I can look into this. As you said, it requires some digging.

[hyang0129]

I could try designing a common test for optimizers.

If I understand correctly, all subclasses of optimizers should have common behaviors (eg. applies gradients, serializes, etc). So given these common behaviours, we could have a test that identifies all the classes in the optimizers subpackage and tests them the same way.

For example, we get a list of optimizers, instantiate them with default parameters, create a dummy model and compile with MSE loss, fit the model for x steps, serialize, deserialize, etc. Would this be a test that the contributor has to add or would it be better if this test automatically finds all optimizers in the subpackage?

[seanpmorgan]

I could try designing a common test for optimizers.

If I understand correctly, all subclasses of optimizers should have common behaviors (eg. applies gradients, serializes, etc). So given these common behaviours, we could have a test that identifies all the classes in the optimizers subpackage and tests them the same way.

For example, we get a list of optimizers, instantiate them with default parameters, create a dummy model and compile with MSE loss, fit the model for x steps, serialize, deserialize, etc. Would this be a test that the contributor has to add or would it be better if this test automatically finds all optimizers in the subpackage?

That would be great! I really like the idea of the test finding and testing all optimizers, but want to make sure the test case has a good UX when reading the test logs (e.g if we could utilize pytest parameterize that'd be great.) Feel free to open a PR and we can iterate on how we enforce the test.