Unity tests no longer show Serial output without verbose option

[torntrousers]

After an updated from v5 to PlatformIO Core, version 6.0.1 Serial output in tests no longer shows unless the verbose option is used.

For recreate see: https://community.platformio.org/t/unity-test-serial-print-output-gone/27800/3?u=torntrousers

[ivankravets]

Hi Anthony,

Thanks for the report. Indeed, this is a feature, not a bug. We don't print all junk from the testing frameworks and provide you by default only valuable information. The --verbose mode is the right solution. However, I agree, it prints now a ton of junk from the previous stages (build/upload).

So, what I propose. Allow passing multiple -v options:

• -v - makes testing verbose
• -vv - makes building and uploading verbose
• -vvv - makes building and uploading much more verbose.

Your thoughts?

[torntrousers]

having all the building etc type messages be moved from -v to -vv / -vvv would definately help.
it wasn't obvious to me how to set verbose mode when running test in vscode, something about project-tasks but it looked a bit complicated.

[torntrousers]

Honestly though, it seems a bit of a pain, why the change?

[ivankravets]

You can override it via pio test -v.

[torntrousers]

Mmm, not yet convinced that this is an improvement over the v5 behaviour. How do I set -v in vscode?

[ivankravets]

Thanks! The configuration option in INI has been removed. However, the multilevel verbosity is available for pio test command. Also, new "Verbose Test" command has been added platformio/platformio-node-helpers@6931b21
Will be a part of the next release of PlatformIO IDE for VSCode.

Meanwhile, you can declare custom target with pio test -v. It will be available in IDE in the Tasks view.

We also plan to create a separate Test View. Please subscribe to this feature platformio/platformio-vscode-ide#3168

[torntrousers]

None of those really seem great though when all i want is to see the test output so i can add debugging output while developing new code, like it used to. I suppose the separate Test View might help, but its not available yet.

[j4m3s]

I've run into the same issue. We write log output to std::cout, and it used to be visible in v5.x. It's now lost/ swallowed in v6.

The pio test -v option in v6.0.2 doesn't help with the noise unfortunately, it still shows all of the tests collected and skipped, and doesn't colour the individual test results etc.

FYI the link included in the comment above doesn't seem to point to anything about the verbosity level?:

A new configuration option is added https://docs.platformio.org/en/latest/projectconf/section_env_test.html#test-verbosity-level

You can also override it via pio test -v.

[ivankravets]

Yes, PlatformIO normalizes test results from multiple testing frameworks to the unified output. Typically, people don't use Unit Testing to debug their applications. If you need custom results output, please override https://github.com/platformio/platformio-core/blob/develop/platformio/test/runners/unity.py#L260

See https://docs.platformio.org/en/latest/advanced/unit-testing/frameworks/custom/index.html

[torntrousers]

Typically, people don't use Unit Testing to debug their applications.

For me, that is the way people use this - write some tests and then using debug output to get the tests passing or to work out whats wrong when some change has inadvertently broken a test.

Personally I prefer the v5 behaviour and wonder wouldn't something like a new "quiet" option have been a less disruptive change for those who don't want to see the output.

If you need custom results output, please override https://github.com/platformio/platformio-core/blob/develop/platformio/test/runners/unity.py#L260

See https://docs.platformio.org/en/latest/advanced/unit-testing/frameworks/custom/index.html

Can you explain that a bit more - how would I do that override?

[ivankravets]

Just follow this example https://docs.platformio.org/en/latest/advanced/unit-testing/frameworks/custom/examples/custom_unity_library.html

platformio.ini

[env:myenv]
test_framework = custom

test_custom_runner.py (put this file to the root of test folder)

import click

from platformio.public import UnityTestRunner

class CustomTestRunner(UnityTestRunner):

    def on_testing_line_output(self, line):
        click.echo(line, nl=False)
        super().on_testing_line_output(line)

[torntrousers]

That gives:

If you don't see any output for the first 10 secs, please reset board (press reset button)

name 'click' is not defined
------------------------------------------------ arduino-esp32mbedtls:* [ERRORED] Took 16.00 seconds ------------------------------------------------

Do I need an import or something for click?

[ivankravets]

Sorry, updated the code above. I didn't test it.

[j4m3s]

Thank you for this @ivankravets. I added an if check so that only strings that aren't internally generated by Unity are printed:

import click

from platformio.public import UnityTestRunner


class CustomTestRunner(UnityTestRunner):

    def on_testing_line_output(self, line):
        if super().parse_test_case(line) is None:
            click.echo(line, nl=False)
        super().on_testing_line_output(line)

[danverstom]

Thank you everyone! I agree that this change of behaviour is a little odd. I was also using unit tests to debug small pieces of code for my project, so I think it is a very valid use case. Thanks for providing a solution, works well for me.

[jhaand]

This solution looks fine to me.

[lemonkey]

Doesn't seem to work for me with platformio 6.1.4 unless I use the TEST_MESSAGE macro (instead of say Serial.print()).

And this is using the "-v" flag when running pio test. I also have the custom runner as mentioned above..

TEST_MESSAGE macro is defined under .pio/libdeps/<your current project env>/Unity/src/unity.h.

#define TEST_MESSAGE(message) UnityMessage((message), __LINE__)

[ivankravets]

Could you share your platformio.ini?

[lemonkey]

It turns out I only have to use TEST_MESSAGE in the main test loop. Serial.print() statements are fine from within the unit tests themselves.