Regression tests (Section 3.3) are not failing with Randoop 4.2.4

[micskeiz]

In recent versions of Randoop, the generated regression tests on the changed version of MyInteger (after gradlew third).

As far as I can see, the issue is that Randoop generates for the original code regression tests, which use most of the time 0 as a primitive value for the int parameter of MyInteger().

Rarely -1 and '#' are also used, but no -1 x -1 multiplication is called, hence the bug is not triggered.

I have not seen 10 or 100 as values, although they are defined in SeedSequences also.

Command used for Randoop as suggested in the tutorial (it was run on Windows 10 with openjdk version "12" 2019-03-19):

java -cp build/classes/java/main;%RANDOOP_JAR% randoop.main.Main gentests --testclass=math.MyInteger --junit-output-dir=src/test/java --output-limit=200

Output of Randoop:

Randoop for Java version "4.2.4, branch master, commit be6dbd3, 2020-07-14".

Will try to generate tests for 1 classes.
PUBLIC MEMBERS=8
Explorer = ForwardGenerator(steps: 0, null steps: 0, num_sequences_generated: 0;
    allSequences: 0, regresson seqs: 0, error seqs: 0=0=0, invalid seqs: 0, subsumed_sequences: 0, num_failed_output_test: 0;
    runtimePrimitivesSeen:38)

Progress update: steps=1, test inputs generated=0, failing inputs=0      (2020-11-26T10:44:36.618545300Z     13MB used)
Progress update: steps=235, test inputs generated=201, failing inputs=0      (2020-11-26T10:44:55.510367600Z     35MB used)
Normal method executions: 661241
Exceptional method executions: 2

Average method execution time (normal termination):      0,000250
Average method execution time (exceptional termination): 0,158
Approximate memory usage 35MB
Explorer = ForwardGenerator(steps: 235, null steps: 34, num_sequences_generated: 201;
    allSequences: 201, regresson seqs: 200, error seqs: 0=0=0, invalid seqs: 0, subsumed_sequences: 0, num_failed_output_test: 1;
    runtimePrimitivesSeen:48)

No error-revealing tests to output.

About to look for failing assertions in 98 regression sequences.

Regression test output:
Regression test count: 98
Writing regression JUnit tests...
Created file C:\code\tutorial-examples\src\test\java\RegressionTest0.java
Created file C:\code\tutorial-examples\src\test\java\RegressionTest.java
Wrote regression JUnit tests.
About to look for flaky methods.

Invalid tests generated: 0

Excerpt from the tests:

        math.MyInteger myInteger1 = new math.MyInteger((int) (short) 0);
        math.MyInteger myInteger3 = new math.MyInteger((int) (short) 0);
        math.MyInteger myInteger4 = myInteger1.multiply(myInteger3);
        math.MyInteger myInteger6 = new math.MyInteger((int) '#');
        math.MyInteger myInteger7 = myInteger4.add(myInteger6);
        math.MyInteger myInteger9 = new math.MyInteger((int) (short) 0);
        math.MyInteger myInteger11 = new math.MyInteger((int) (short) 0);
        math.MyInteger myInteger12 = myInteger9.multiply(myInteger11);

Interestingly, if I add a TestValue file with this content:

  package math;
  import randoop.*;

  public class TestValueExamples {

    @TestValue
    public static int i = -333;

  }

and use the command (with classlist.txt having MyInteger and TestValueExample):

java -cp build/classes/java/main:%RANDOOP_JAR% randoop.main.Main gentests --classlist=classlist.txt --junit-output-dir=src/test/java --output-limit=200

then the tests contain mostly -1 (but no -333), and the bug is triggered:

        math.MyInteger myInteger1 = new math.MyInteger((int) (byte) -1);
        math.MyInteger myInteger3 = new math.MyInteger((int) (byte) -1);
        math.MyInteger myInteger4 = myInteger1.multiply(myInteger3);
        math.MyInteger myInteger6 = new math.MyInteger((int) (byte) -1);
        math.MyInteger myInteger8 = new math.MyInteger((int) (byte) -1);
        math.MyInteger myInteger9 = myInteger6.multiply(myInteger8);

Are there any parameters to control how Randoop uses seed values?

[leventeBajczi]

This workaround works well, but to use it, a dependency has to be added for the randoop jar file. build.gradle needs to be modified the following way (given that $RANDOOP_JAR holds the path to the jar file):

// In this section you declare the dependencies for your production and test code
dependencies {
    // The production code uses the SLF4J logging API at compile time
    implementation 'org.slf4j:slf4j-api:1.7.21'

    // Declare the dependency for your favourite test framework you want to use in your tests.
    // TestNG is also supported by the Gradle Test task. Just change the
    // testCompile dependency to testCompile 'org.testng:testng:6.8.1' and add
    // 'test.useTestNG()' to your build script.
    testImplementation 'junit:junit:4.12'
    implementation files("$System.env.RANDOOP_JAR")
}