tests/plugins: refactor

[GaetanLepage]

In the existing version of the tests, all the files from the tests/plugins/ folder are listed and imported.
This was preventing us from structuring the content of this directory.
I removed this in favor of hard-coded imports (i.e. we explicitly list the test files we wish to import).
This allows for a conventional way of ordering the test cases.

I suggest that we keep the structure of the tests/plugins/ folder exactly the same as the one of the plugins/ folder.

[traxys]

There is another option, we can have default.nix files for each subdirectories that would perform exactly the same work as the root file, i.e: Concat all the subfiles in a list with the path added to the name.

I don't have any problems with the approach in this PR, but I don't know what you think @pta2002

[pta2002]

You can have the automatic listing of the files and still support nesting. Just takes a bit more work, but not too much. I'd personally be more in favor of this since it avoids having to change the tests file every time

[GaetanLepage]

You can have the automatic listing of the files and still support nesting. Just takes a bit more work, but not too much. I'd personally be more in favor of this since it avoids having to change the tests file every time

Hmm I agree that it is feasible. I actually have already taken a look at this before submitting this PR.
I feel like the explicit listing is more simple-stupid but is still satisfying. If a PR adds a test, we just have to include it in the main file.
For me it is exactly the same process as when we add a new plugin: We have to add the file of the plugin and mention it in the plugins/default.nix file as well.

Anyway, if you feel like it is indeed better to support automatic nesting of nested folders, I can understand :)

[pta2002]

I think specifically in this case it's better than in the case of plugins. Picture this:

• You add a new test, by creating the file
• You submit a PR for it

Now, imagine that you had to manually list it. What would the results be? The test just straight up would not run, because it's easy to forget to add to it, and it's really easy to miss without creating some automated rule for it... and at that point you might as well just auto-import it ;)

This isn't as much of an issue with plugins because ideally you'd be testing it anyway and you'd catch it if you forgot to import it.

[GaetanLepage]

I think specifically in this case it's better than in the case of plugins. Picture this:

* You add a new test, by creating the file

* You submit a PR for it

Now, imagine that you had to manually list it. What would the results be? The test just straight up would not run, because it's easy to forget to add to it, and it's really easy to miss without creating some automated rule for it... and at that point you might as well just auto-import it ;)

This isn't as much of an issue with plugins because ideally you'd be testing it anyway and you'd catch it if you forgot to import it.

I agree. It makes sense indeed !
I will take a look at that soon :)

[GaetanLepage]

Hi @pta2002 !

@traxys and I have been working to make this new proposal.
We now have a much clearer test cases management.

Changes:

1. Everything regarding testing is now in the tests/ directory. In particular, the code that was previously in lib/check.nix has been moved there.
2. The folder tests/test-sources/ contains all and only the source of the test cases.
3. The nix files at the root of the tests/ directory contain the necessary functions to gather the test cases and convert them to derivations. Btw, the gathering is automatically looking for all the test cases recursively in tests/test-sources/
4. The previous tests/flake.nix has been deleted. The test cases that were living in it were moved to the tests/test-sources/ directory.

We are looking forward to your review,

Best :)

[pta2002]

This looks very good, thanks for the work in this, and sorry for taking so long to look at it!

[GaetanLepage]

This looks very good, thanks for the work in this, and sorry for taking so long to look at it!

Thanks for the feedback :)