Feature Request: Implementing custom messages for REQUIRE/CHECK macros.

Description

This is a feature Boost.Test has (BOOST_<level>_MESSAGE). It would be very nice if we can define custom error message if any specific test fails. A use-case is as follows:

int minutes = 100;
REQUIRE_MESSAGE( 0 <= minutes <= 59 , "Minutes " << minutes << " outside allowed range.");

Output:

Example.cpp:X: FAILED:
Minutes 100 outside allowed range.

Here is how u can extend Catch to have these macros:

#define CHECK_MESSAGE(cond, msg) do { INFO(msg); CHECK(cond); } while((void)0, 0)
#define REQUIRE_MESSAGE(cond, msg) do { INFO(msg); REQUIRE(cond); } while((void)0, 0)

Thank you for you reply. I know there are ways of rolling my own macros, but it is still better if this feature is built-in.

Also if I run my test command with -s option, the message shows up, even if the message only talks about how a specific test case fails.

You might want to check out matchers. They are bit of an overkill for simple expressions like in the example, but those should be self-documenting anyway. For more complex tests they let you introduce any amount of logic needed for proper testing and stringification.

Yes I sometimes use matchers to fulfill similar function. However like you said this is an overkill and not as convenient as having a built-in macro. Using a CHECK_MESSAGE and a REQUIRE_MESSAGE macro surely require a lot less lines of codes than implementing custom matcher classes.

Another drawback with the matcher classes is that the value under test must preceded the string output, I don't have the flexibility of putting the values in the middle of the string output, whereas this can be easily done with a CHECK_MESSAGE or REQUIRE_MESSAGE macro.

The trouble with specific examples is they may not convey the general idea of why you want something.
So what I say next is on the basis that it may prompt you to give another example that forces us to generalise - in the true spirit of TDD :-)

For your example I would write it something more like:

TEST_CASE("minutes") {
    int minutes = 100;
    SECTION( "Minutes must be within range" ) {
        REQUIRE( minutes >= 0 );
        REQUIRE( minutes <= 59 );
    }
}

The output, on failure, is then:

-------------------------------------------------------------------------------
minutes
  Minutes must be within range
-------------------------------------------------------------------------------

[..]

  REQUIRE( minutes <= 59 )
with expansion:
  100 <= 59

Which would seem to have all the information you need. Granted it's not organised exactly how you would like - but I think it's usually good enough. We don't want to spend our timing optimising for output formatting on the failing case (which we should, hopefully, rarely see) - so I've tried to design things so that you get rich enough feedback with minimum effort.

So, now, you can either disagree that this is enough in this case (and if you have thoughts there I'm still interested to hear them - whether or not we end up changing anything) - or you might concede on this example and think of another one that better teases out what you want.

Your move :-)

Thank you for your reply. Yes I think my previous example was indeed too trivial and does not justify an extra macro. However custom messages are really helpful sometimes when we need to peek into the internal state of the program to see why a particular test fails. A use case for this new macro is as follows:

TEST_CASE("Widget") {
    for (int ctorParam1 = 0; ctorParam1 < 10; ++ctorParam1) {
        for (int ctorParam2 = 0; ctorParam2 < 10; ++ctorParam2) {
            Widget w(ctorParam1, ctorParam2);
            w.doSomething();
            REQUIRE_MESSAGE(w.valueX() > 0, "The X value of Widget is invalid for when ctorParam1 = "
                                            << ctorParam1 << " and ctorParam2 = " << ctorParam2
                                            << ". The internal state of Widget is " << w.someInternalParameter() << ".");
        }
    }
}

There are alternative ways of getting the internal state of the program when a test case fails, but I think using a custom message for the REQUIRE/CHECK macros is by far the most convenient way.

Ok, but what's wrong with?:

TEST_CASE("Widget") {
    for (int ctorParam1 = 0; ctorParam1 < 10; ++ctorParam1) {
        for (int ctorParam2 = 0; ctorParam2 < 10; ++ctorParam2) {
            CAPTURE( ctorParam1 );
            CAPTURE( ctorParam2 );
            Widget w(ctorParam1, ctorParam2);
            w.doSomething();
            INFO( "The internal state of Widget is " << w.someInternalParameter() );
            REQUIRE(w.valueX() > 0);
        }
    }
}

Yes there are ways of capturing the internal state of the program already even without this extra macro. I just think it is more convenient to combine the custom message and the assertion macros into a single macro. Also the users have more freedom if we can customize the fail message.

You can format INFO to look exactly like your REQUIRE_MESSAGE output - it's just a bit more fiddly (yet still less typing than REQUIRE_MESSAGE - less characters in INFO() than _MESSAGE :-) )
The only real difference is that the message goes before the assertion.
But it's more flexible because you can have several of them. In my example I put the two captures earlier. I didn't have to, but because I could it means that if the constructor of Widget throws, or the call to doSomething(), then those values will be logged - even though we haven't got to the assertion yet. I used CAPTURE because that automatically logs the name the variable and it's value - saving some redundancy - but does mean you may have to type more - I could have just put it all in an INFO)

So, in summary, using INFO is:

a) less typing
b) more flexible
c) allows more state to be captured at more times

with the only "downside" being that it appears before the assertion instead of after it.

That sounds like a good set of trade-offs to me :-)

So is it just an aesthetic thing, or do you still think REQUIRE_MESSAGE is more convenient?

Thank you for your detailed explanation. Now thinking about it more, the difference is just an aesthetic thing. The extra macro does not provide as much convenience as I previously thought.

there is still nothing close to python's

assert expression, "Error message"

I would like this feature and the macros above don't provide the desired behavior (the message is printed even if the assert passes). Can we re-open the issue?

I think that a message that should be printed when an assertion fails to better explain what failed is semantically different than printing all gathered information messages when the same assertion fails. To explain it better with a Catch2 example, this looks a bit ugly to me:

INFO("Other unrelated info")
[...]
INFO("Something went wrong")
REQUIRE(false);

GoogleTest seems to identity the use case better and supports this syntax:

EXPECT_TRUE(false) << "Something went wrong";

This look semantically more correct to me and offer the opportunity also to colorize/highlight the output in a different way for the failed assertion message. I'm very new to Catch2 and I don't understand if Catch2 has something similar. If it has please point it to me, otherwise the I also ask for this issue to be re-opened and think about implementing something similar to what GoogleTest does.

Navigation Menu

Repository navigation

Feature Request: Implementing custom messages for REQUIRE/CHECK macros. #929

Description

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Feature Request: Implementing custom messages for REQUIRE/CHECK macros. #929

Description

Description

Activity

onqtam commented on Jun 15, 2017

haozhangphd commented on Jun 15, 2017

horenmar commented on Jun 15, 2017

haozhangphd commented on Jun 15, 2017

philsquared commented on Aug 11, 2017

haozhangphd commented on Aug 11, 2017

philsquared commented on Aug 11, 2017

haozhangphd commented on Aug 11, 2017

philsquared commented on Aug 11, 2017

haozhangphd commented on Aug 12, 2017

eugene-kulak commented on Jan 10, 2021

jcummings2 commented on Jun 29, 2021

ceztko commented on Nov 27, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions