Running unit testing with no tests should output a warning if the build fails

[nomis]

Configuration

Operating system: Ubuntu 18.04

PlatformIO Version (platformio --version): 6.0.1

Description of problem

With 6.0.0 all test directories must now begin with test_ (#4135) but there's no warning when PlatformIO tries to build the test application without any test code. Tests fail to build with no explanation of why.

There's also no way to view the older documentation (for 5.x) on https://docs.platformio.org/en/latest/advanced/unit-testing/structure.html.

Steps to Reproduce

1. Have a test directory that doesn't contain any test_ directories.
2. Run platformio test

Actual Results

Failure to build because of undefined references to main(), setup(), loop() or other functions used by the project under test that don't exist because there is no test code.

Expected Results

Failure to build with an explanatory warning message that the project appears to have no tests (other than the * test) and link to the new documentation.

[nomis]

The "FULLY BACKWARD COMPATIBLE" notice should be amended too, because it's not...

[ivankravets]

Thanks for the report. Please re-test with pio upgrade --dev.

Does it work now?

[nomis]

No, the test build proceeds despite there being no tests so it fails and doesn't get a chance to display that message:

$ git clone https://github.com/nomis/mcu-uuid-console
$ cd mcu-uuid-console
$ git checkout ecf24c551e22a15a80106ba658811d4c0e0bf00e
$ cd test/native
$ platformio --version
PlatformIO Core, version 6.0.2b1
$ platformio test
Warning! `src_build_flags` configuration option in section [env:native] is deprecated and will be removed in the next release! Please use `build_src_flags` instead
Warning! `test_build_project_src` configuration option in section [env:native] is deprecated and will be removed in the next release! Please use `test_build_src` instead
Verbosity level can be increased via `-v, -vv, or -vvv` option
Collected 1 tests

Processing * in native environment
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Building...
Warning! `src_build_flags` configuration option in section [env:native] is deprecated and will be removed in the next release! Please use `build_src_flags` instead
Warning! `test_build_project_src` configuration option in section [env:native] is deprecated and will be removed in the next release! Please use `test_build_src` instead
/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/Scrt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
.pio/build/native/src/shell.o: In function `uuid::console::Shell::check_idle_timeout()':
shell.cpp:(.text+0x3d9): undefined reference to `uuid::get_uptime_ms()'
.pio/build/native/src/shell.o: In function `uuid::console::Shell::loop_password()':
shell.cpp:(.text+0x7bb): undefined reference to `uuid::get_uptime_ms()'
.pio/build/native/src/shell.o: In function `uuid::console::Shell::loop_blocking()':
shell.cpp:(.text+0x89f): undefined reference to `uuid::get_uptime_ms()'
.pio/build/native/src/shell.o: In function `uuid::console::Shell::loop_delay()':
shell.cpp:(.text+0x8f8): undefined reference to `uuid::get_uptime_ms()'
shell.cpp:(.text+0xa09): undefined reference to `uuid::get_uptime_ms()'
.pio/build/native/src/shell.o:shell.cpp:(.text+0x11eb): more undefined references to `uuid::get_uptime_ms()' follow
collect2: error: ld returned 1 exit status
*** [.pio/build/native/program] Error 1
Building stage has failed, see errors above. Use `pio test -vvv` option to enable verbose output.
------------------------------------------------------------------------------------ native:* [ERRORED] Took 0.57 seconds ------------------------------------------------------------------------------------

=================================================================================================== SUMMARY ===================================================================================================
Environment    Test    Status    Duration
-------------  ------  --------  ------------
native         *       ERRORED   00:00:00.573
================================================================================== 1 test cases: 0 succeeded in 00:00:00.573 ==================================================================================

[ivankravets]

Sorry, should be finally fixed. Please re-test with pio upgrade --dev.

[nomis]

It works now:

Error: Nothing to build. Please put your test suites to the '/home/simon/src/mcu-uuid-console/test/native/test' folder
Building stage has failed, see errors above. Use `pio test -vvv` option to enable verbose output.

[ivankravets]

Thanks for the reports and help with testing! The final release of PlatformIO Core 6.0.2 is planned for Monday/Tuesday