Some meaningful test is expected for a regexp

[Deuqz]

Description

No strings are generated for a regexp pattern to check.

To Reproduce

1. IntelliJ IDEA is opened with UTBot plugin installed
2. Project with JDK 11 and a Java class is opened
3. Add the following code:

public boolean bar(@NotNull String s) {
    return s.matches("[a-zA-Z]*");
}

4. Use UTBot plugin to generate tests

Expected behavior

Tests with strings matching and not matching regexp are supposed to be generated.

Actual behavior

There is only one test with the regexp:

@Test
@DisplayName("bar: s = string -> return false")
public void testBarReturnsFalseWithNonEmptyString() {
    Bor1 bor1 = new Bor1();

    boolean actual = bor1.bar("[a-zA-Z]*");

    assertFalse(actual);
}

Environment

JDK: correto-11
Language level: 11
Intellij IDEA: 2021.1.4
Operating system: Windows 10

[alisevych]

Additional context

If Fuzzing is turned off (Settings -> Tools -> UTBot, set Code analysis to the rightmost position)
Symbolic execution generates a test with a string that matches the regexp pattern:

@Test
    @DisplayName("bar: StringMatches -> return s.matches(\"[a-zA-Z]*\")")
    public void testBar_StringMatches() {
        RegexpExample regexpExample = new RegexpExample();
        String s = "ORoeTJjstPwXwpdofuShKZSihBScKn";

        boolean actual = regexpExample.bar(s);

        assertTrue(actual);
    }

But still there is a negative test also expected for such cases.

[dtim]

It is indeed a conceptual issue.

Currently UnitTestBot generates regression test suites and tries to maximize the coverage. The single "positive" test covers all the instructions in the method body, so we don't need any additional tests (e.g., a "negative" test where the input string does not match the regex) as they would not improve the coverage.

Note: if we generate a test suite for a method that calls bar() and branches on its result, the symbolic engine will generate two input strings, one matching the regex, and another not matching it.

On the other hand, in a functional test suite I'd prefer to have tests for both potential outcomes for bar() regardless of the context where it is called.

Functional tests check the compliance of code to a set of predefined requirements. For example, we can have an API specification that explicitly says that bar() may return true if its input is an empty string or consists only from English letters regardless of their case, or false in any other case. We would like to have a test suite that checks both of these statements, even that any one of them achieves full coverage.

This issue seems to concern with functional testing, that is not currently supported by UnitTestBot. Functional test generation is a valid goal, and it should probably be addressed. The main limitation just now is that we don't currently have any way to obtain functional requirements, and to use them to guide the test generation. It seems an interesting research topic.

[alisevych]

As discussed with @denis-fokin the expected behavior for such a case is to create separate tests - both for true and false outcomes. Support of boolean functions is planned to take into work not immediately but in the nearest future.
Functional tests generation for other numerable results (Enums) needs to be considered in future too.