Initial section on testing

README.md

@@ -34,3 +34,5 @@ The Developers' Guide is continuously updated and there are several parts still
 * Text about JCK
 * Text about backports
 * Text about JBS, triage, priorities, status
+* List JTReg `@key` conventions for different areas
+* Document best practices around TEST.properties usage. See [PR#30](https://github.com/openjdk/guide/pull/30#issuecomment-714589551)

src/index.md

@@ -333,6 +333,141 @@ For the purposes of brevity this document will use the term "bug" to refer to bo
 
 _Congratulations!_ Your changeset will now make its way towards a promoted build. When the changeset becomes part of a promoted build, the bug's "Resolved in Build" will have a value of \"b\[1-9\]\[0-9\]*\" to indicate the build number.
 
+# Testing Changes
+
+In addition to your own Java applications, OpenJDK have support for two test frameworks, JTReg and GTest. JTReg is a Java regression test framework that is used for most of the tests that are included in the OpenJDK source repository. The Google Test (GTest) framework is intended for unit testing of the C++ native code. Currently only JVM testing is supported by the GTest framework. Other areas use JTReg for unit testing of C++ code.
+
+This section provides a brief summary of how to get started with testing in OpenJDK. For more information on configuration and how to use the OpenJDK test framework, a.k.a. "run-test framework", see [`doc/testing.md`](https://github.com/openjdk/jdk/blob/master/doc/testing.md).
+
+In general all changes should come with a regression test so if you're writing product code you should also be writing test code. There are a few examples where it doesn't make sence to write an explicit regression test. These should be tagged in JBS with one of the [noreg-labels](#noreg).
+
+A few key items to think about when writing a regression test:
+
+* A regression test should execute fast - a few seconds at most
+* The test should only test the desired functionality - if you have several features to test, write more tests
+* The test should pass reliably on all supported platforms - watch out for platform-specific differences such as path separators
+* Binary files should not be checked in, if your test needs to use one, the test should create it in some fashion
+* Avoid shell scripts and relying on external commands as much as possible
+
+The JTReg documentation has a section on [how to write good JTReg tests](https://openjdk.java.net/jtreg/writetests.html).
+
+## JTReg
+
+In-depth documentation about the JTReg framework is found here: [JTReg harness](https://openjdk.java.net/jtreg/). JTReg itself is available in the [Code Tools Project](https://openjdk.java.net/projects/code-tools/).
+
+Below is a small example of a JTReg test. It’s a clean Java class with a main method that is called from the test harness. If the test fails we throw a RuntimeException. This is picked up by the harness and is reported as a test failure. Try to always write a meaningful message in the exception. One that actually helps with understanding what went wrong once the test fails.
+
+    /*
+     * @test
+     * @summary Make sure feature X handles Y correctly
+     * @run main TestXY
+     */
+    public class TestXY {
+        public static void main(String[] args) throws Exception {
+            var result = X.y();
+            if (result != expected_result) {
+                throw new RuntimeException("X.y() gave " + result + ", expexted " + expected_result);
+            }
+        }
+    }
+
+This example only utilizes three JTReg specific tags, `@test`, `@summary`, and `@run`. `@test` simply tells JTReg that this class is a test, and `@summary` provides a description of the test. `@run` tells JTReg how to execute the test. In this case we simply tell JTReg to execute the main method of the class `TestXY`. `@run` is not strictly necessary for JTReg to execute the test, an implicit `@run` tag will be added if none exists. However, for clarity and in order to avoid bugs it's recommended to always explicitly use the `@run` tag.
+
+There are several other tags that can be used in JTReg tests. You can for instance associate the test with a specific bug that this test is a regression test for.
+
+    @bug 7000001
+
+Or you can specify a number of requirements that must be fulfilled for JTReg to execute the test.
+
+    @requires docker.support
+    @requires os.family != ”windows”
+    @requires os.maxMemory > 3G
+    @requires os.arch=="x86_64" | os.arch=="amd64"
+
+You can also specify if the test requires specific modules, and you can specify command line flags and run the test in several different ways.
+
+    @modules java.base/jdk.internal.misc
+    @run main/othervm -Xmx128m TestXY
+
+Note that you can have several `@run` tags in the same test with different command line options.
+
+JTReg also have support for labeling tests with keys using the `@key` tag. These keywords can then be used to filter the test selection. For instance if you have a UI test which needs to display a window you'll want to make sure the test harness doesn't try to run this test on a system which doesn't support headful tests. You do this by specifying
+
+    @key headful
+
+Another example is `@key randomness` that should be used to indicate that a test is using randomness - i.e. is intentionally non-deterministic.
+
+There are many other keywords in use and their usage may differ between areas in the JDK. Make sure you understand the conventions for the particular area you are testing since these are just examples.
+
+The [JTReg documentation](https://openjdk.java.net/jtreg/) provides information on many more tags like these.
+
+The [compiler group](https://openjdk.java.net/groups/compiler/) has a section in their wiki with [Guidelines for "langtools" tests](https://openjdk.java.net/groups/compiler/tests.html).
+
+### Running OpenJDK JTReg Tests
+
+When configuring the OpenJDK build you can tell it where your JTReg installation is located. When providing this information you can later run `make run-test` to execute JTReg tests.
+
+    sh ./configure --with-jtreg=/path/to/jtreg
+    make run-test TEST=tier1
+
+In the OpenJDK source tree you can find a directory called `test`. There are a large number of tests in this directory that are written to be used with JTReg.
+
+    make run-test TEST=test/jdk/java/lang/String/
+
+You can also run JTReg without invoking make. In this case you’ll need to tell JTReg which JDK to test.
+
+    jtreg -jdk:/path/to/jdk /path/to/test
+
+## GTest
+
+As mentioned the Google test framework is mainly used for C++ unit tests. There are several of these in the `test/hotspot` directory. Currently, only the C++ code in the JVM area is supported by the OpenJDK GTest framework. The tests can be run without starting the JVM, which enables testing of JVM data structures that would be fragile to play with in a running JVM.
+
+    static int demo_comparator(int a, int b) {
+      if (a == b) {
+        return 0;
+      }
+      if (a < b) {
+        return -1;
+      }
+      return 1;
+    }
+
+    TEST(Demo, quicksort) {
+      int test_array[] = {7,1,5,3,6,9,8,2,4,0};
+      int expected_array[] = {0,1,2,3,4,5,6,7,8,9};
+
+      QuickSort::sort(test_array, 10, demo_comparator, false);
+      for (int i = 0; i < 10; i++) {
+        ASSERT_EQ(expected_array[i], test_array[i]);
+      }
+    }
+
+`ASSERT_EQ` is one example of an assertion that can be used in the test. Below are a few other examples. A full list is found in the [Google Test Documentation](https://github.com/google/googletest/blob/master/googletest/docs/primer.md).
+
+    ASSERT_TRUE(condition);
+    ASSERT_FALSE(condition);
+    EXPECT_EQ(expected, actual);
+    EXPECT_LT(val1, val2);
+    EXPECT_STREQ(expected_str, actual_str);
+
+`ASSERT` is a fatal assertion and will interrupt execution of the current sub-routine. `EXPECT` is a nonfatal assertion and will report the error but continues to run the test. All assertions have both an `ASSERT` and an `EXPECT` variant.
+
+For more information on how to write good GTests in HotSpot, see [`doc/hotspot-unit-tests.md`](https://github.com/openjdk/jdk/blob/master/doc/hotspot-unit-tests.md).
+
+### Running OpenJDK GTests
+
+When configuring the OpenJDK build you can tell it where your GTest installation is located. Once configured, use make to run GTests.
+
+    sh ./configure --with-gtest=/path/to/gtest
+    make test TEST=gtest
+
+You can also use a regular expression to filter which tests to run:
+
+    make test TEST=gtest:code.*:os.*
+    make test TEST=gtest:$X/$variant
+
+The second example above runs tests which match the regexp `$X.*` on a specific variant of the JVM. The variant is one of client, server, etc.
+
 # Producing a Changeset
 
 This section is confined to the actual Mercurial mechanics required to produce a changeset:

src/testingChanges.md

@@ -1,19 +1,3 @@
-% Testing Changes
+% Page moved
 
-::: {.NavBit}
-[« Previous](reviewBodies.html) • [TOC](index.html) • [Next »](jckAcquisition.html)
-:::
-
-This section will describe the need for tests and will provide an engineering
-perspective to the different kinds of test suites, including regression tests
-as run by the [jtreg harness](../jtreg/) and JCK
-tests. Characteristics of a good regression test will be provided.
-
-<!--
- See also the <a href="https://openjdk.java.net/groups/quality/">OpenJDK
-Quality Group</a>. 
--->
-
-::: {.NavBit}
-[« Previous](reviewBodies.html) • [TOC](index.html) • [Next »](jckAcquisition.html)
-:::
+The OpenJDK Developers' Guide has been merged into a single document. Please update your links to point to the new location: [OpenJDK Developers' Guide](index.html)