Add tests with pytest

[huguesdevimeux]

So I created a bunch of tests for pytest. To run them, pytest command is enoug
I tried to test pretty much all of the features that manim currently has. Don't hesitate to tell if some miss.

Functioning :

See #34 and #97.
For performances purpose it just compares the last frame with a previously-rendered one (and not a frame every n seconds, as it was mentioned in #34. On my computer, it takes around 30 seconds.

Adding a new test

I created the function utils.set_test that is intended to be used when we add a new feature.
I think that we should create a section in the wiki to explain how to create a new test and mention it in #127.

For the moment I marked this PR as a draft since it uses some stuff (see conftest.py) that will be completely rewritten when #98 is merged.

closes #28 closes #34 closes #97

[hadichaudhri]

Same thing (as far as DRY goes) for Windows here. We probably want to use a .bat script to set up the environment.

[hadichaudhri]

This comment was supposed to be 3 lines up in before-install, but you get the gist.

[huguesdevimeux]

As a lot of things depend on when this PR is merged, I propose to merge it now.

All of the suggestions that had been said here were all very good and I'm working on them, but they don't concern the core of the tests. I will do another PR.

[PgBiel]

what are the tests_cache and tests_data directories for? Shouldn't Travis or whatever automatically generate those folders? Then they shouldn't be in the repo and should be in .gitignore
Also, should we follow guidelines and also add typings to the testing_utils.py file?

[huguesdevimeux]

what are the tests_cache and tests_data directories for? Shouldn't Travis or whatever automatically generate those folders? Then they shouldn't be in the repo and should be in .gitignore
Also, should we follow guidelines and also add typings to the testing_utils.py file?

These files are the whole point of the tests. tests_data contains the previously rendered frames that serve for the comparison (if the frames match, then all good and if not, a thing had been changed). Thus, in essence, it can't be generated by Travis itself.
tests_cache is the cache used sometimes in the tests. In that case, for the moment, only TexTest uses it, because the purpose of this test is not to test the caching functionality of manim. That is tested somewhere else, in an end-to-end test.

About typing stuff, maybe you're right but that's not the correct PR to do that, I think.

[PgBiel]

These files are the whole point of the tests. tests_data contains the previously rendered frames that serve for the comparison (if the frames match, then all good and if not, a thing had been changed). Thus, in essence, it can't be generated by Travis itself.

I see. In this case, the files should be stored elsewhere in my opinion, such as in a separate branch or site or whatever, then we should have Travis fetch those.

tests_cache is the cache used sometimes in the tests. In that case, for the moment, only TexTest uses it, because the purpose of this test is not to test the caching functionality of manim. That is tested somewhere else, in an end-to-end test.

Same comment as above ^

About typing stuff, maybe you're right but that's not the correct PR to do that, I think.

You're probably right, but what about documentation following the guidelines?

[huguesdevimeux]

I see. In this case, the files should be stored elsewhere in my opinion, such as in a separate branch or site or whatever, then we should have Travis fetch those.

To be honest, I don't think this is a good idea. For two main reasons :

• This seems to be complicated to set up and .. maybe it is not worth it. But I'm not an expert, so maybe I'm wrong here.
• We have to think of when someone adds new functionality or modify one, one must be able to add/modify a frame contained in test_data. With this system, this would make the thing not simple.

But why do you want this? Is there like a convention or something? Or is it just cleaner ?

For documentation following the guidelines, you are totally right. I will commit asap.

[PgBiel]

But why do you want this? Is there like a convention or something? Or is it just cleaner ?

I simply thought it would be cleaner in a way, given that we will all be cloning this repo (+ users downloading, soon) so I thought it would be a good idea to not have a lot of otherwise irrelevant "bloat" yk. There probably are alternatives here, but we gotta think this through

[huguesdevimeux]

Then if you find a way that would be nice, I would suggest doing a new PR

[leotrs]

The test cache files are an integral part of the testing system we have setup. I think they should be in this repo, in the tests directory.

@huguesdevimeux you do raise a good point. Most of our comments were not about the core testing system that this is setting up. Can I ask you to please extract the standing discussion points and making separate issues for those that warrant it?

If you think this is ready to be merged, then say so and I'll merge it.