Added doctest unit test framework

[RevoluPowered]

This uses doctest to implement unit testing for the core of godot.

This includes the following:

• a very quick test runner of a single test by running the godot executable.
• the third party library with license file (MIT) and appropriate documentation in the third party folder.
• modifications to SCSub file to make tests part of main executable as linking tests statically can be hard (need a second opinion on this approach too)

Benefits

• horrible regressions which we don't want to come back can have unit tests!
• Simple to use (https://github.com/godotengine/godot/pull/40148/files#diff-b3c825b8305cec3713d66729d1efcd90)
• No boilerplate
• headless or non headless possibility
• CI integration (https://github.com/onqtam/doctest/blob/master/doc/markdown/reporters.md)
  • has all the weird and wonderful reporters for various CI tools
  • CI can be run with headless tests like for string, basis and math checks we do not require a renderer.
• Less files
• follows KISS principle and supports a lot of other useful features https://blog.jetbrains.com/rscpp/better-ways-testing-with-doctest/
• less code per test (because you don't need to copy paste a test harness)

Current test runner can be executed by running ./bin/godot.x11.tools.64 --test string, or by specific test name

Demo with ./godot --test argument

Demo with backward compatibility with other core tests --test math

Articles of supporting info:

• https://github.com/onqtam/doctest/blob/master/doc/markdown/benchmarks.md
• https://blog.jetbrains.com/rscpp/better-ways-testing-with-doctest/

[RevoluPowered]

#30743 - updated doctest for 4.0 fixed conflicts

[akien-mga]

Current test runner can be executed by running ./bin/godot.x11.tools.64 --test doctest

How would future tests be handled? Would it be --test doctest as an entry point to run all tests, or can we keep using --test string instead, as it's not particularly relevant what framework is used when calling the test?

Once this is merged, I think we should port all other tests to doctest and document the workflow.

[akien-mga]

Some style issues: https://travis-ci.org/github/godotengine/godot/jobs/705243011

Also make sure to use braces around single-line if or for blocks.

You lost some of @aaronfranke's changes in test_string while rebasing, so it fails building on Clang: https://travis-ci.org/github/godotengine/godot/jobs/705243015
(Be sure to keep the MinGW-specific hack too.)

[RevoluPowered]

@akien-mga I have applied the requested changes, but made a slight improvement to the mingw specific section.

I used PRId64 (inttypes.h) to correct this issue as it handles the platform specific issue here for us, but I need to test this on mingw, so will do this in a bit and if it works would it be okay to keep this?

The fix is based on https://stackoverflow.com/questions/9225567/how-to-print-a-int64-t-type-in-c this article, just so we don't have to duplicate code or do extra casting.

Also just applied the fix you mentioned regarding the {} parenthesis

[RevoluPowered]

Current test runner can be executed by running ./bin/godot.x11.tools.64 --test doctest

How would future tests be handled? Would it be --test doctest as an entry point to run all tests, or can we keep using --test string instead, as it's not particularly relevant what framework is used when calling the test?

Once this is merged, I think we should port all other tests to doctest and document the workflow.

I will write a quick fix for this now, to make it so

[RevoluPowered]

@akien-mga OK all points raised should now be sorted, I expanded the run functionality to make it more usable for regression testing too, it's super simple lets you run whatever specific test you would like too, also leaves legacy compatibility in.

demo gif

@aaronfranke can you possibly take a look at the fix I applied here, I think it could be simpler for godot to use this instead, it should remove the need for the preprocessor.

1cd932d

reference: https://stackoverflow.com/questions/9225567/how-to-print-a-int64-t-type-in-c

[RevoluPowered]

Nice work, I'll try to test it today.

@vnen @Faless You might want to take a look, and see if you like the new format to write some tests for GDScript and networking respectively.

I had to remove the legacy tests unfortunately so we need to port them

[akien-mga]

I had to remove the legacy tests unfortunately so we need to port them

That's fine, I don't think there was much to salvage in most legacy tests in the first place.

It's better to start cleanly with the new framework, and then port and modernize the tests which were up-to-date enough with the Godot API (like hashmaps and AStar tests).

[RevoluPowered]

That's fine, I don't think there was much to salvage in most legacy tests in the first place.

It's better to start cleanly with the new framework, and then port and modernize the tests which were up-to-date enough with the Godot API (like hashmaps and AStar tests).

Cool, so I left the real files in main/tests/ but they do nothing at the moment to clarify that I didn't remove the actual files, just the legacy test runner, so whoever ports them its simpler, I will try to help with this effort and get them across but need to do this after merge, basis, fbx and math will probably be the ones I port after merge

[RevoluPowered]

Possibly we need some system for measuring the code coverage later on too.

[akien-mga]

Needs a rebase after #40556.

[RevoluPowered]

Needs a rebase after #40556.

Done, I ensured no extra whitespace.

[akien-mga]

Suggested change

	# execute unit tests for the editor
	# Execute unit tests for the editor

Consistency! :)

[RevoluPowered]

For Devs:
To use the unit testing PR you can include the header from thirdparty/doctest/doctest.h it's single header only.

In order to make sure your test is built, once you write your test include the header for it in main/tests/test_main.cpp this ensures it is built as part of the same static linkage to godot's test harness.

To run tests you can use ./godot --test after running this will mean that you no longer can use godot's arguments, you must now use doctest arguments.

The basics:

TEST_CASE("some regression test") {
    CHECK(something_worked());
}

Reference guide
https://github.com/onqtam/doctest

Things which wont work

// by design errors a check should be a SINGULAR check.
CHECK(something && something_else);
CHECK(something || something_else);

[akien-mga]

Thanks a ton!

[aaronfranke]

--test needs to be added to the --help page. Currently --help | grep test returns nothing.

[Calinou]

@aaronfranke As of 6831da6, I can see it at the end of the text returned by --help:

--test [--help]                  Run unit tests. Use --test --help for more information.

[Xrayez]

Yes, this was done in #40832, the old tests are inaccessible but are still there, but some of those are resurrected now: #41355. See preliminary documentation about tests and those additional test tools in godotengine/godot-docs#4017.