[Unix|Windows|AllArch] Unify exclusion list and enable runtest.py

Documentation/building/test-configuration.md

@@ -39,15 +39,17 @@ Test cases are categorized by priority level. The most important subset should b
     ```
 * Disable building of a test by conditionally setting the `<DisableProjectBuild>` property.
 	* e.g. `<DisableProjectBuild Condition=" '$(Platform)' == 'arm64' ">true</DisableProjectBuild>`
+* Exclude test from GCStress runs by adding the following to the csproj:
+    * `<GCStressIncompatible>true</GCStressIncompatible>`
+* Exclude test from JIT stress runs runs by adding the following to the csproj:
+    * `<JitOptimizationSensitive>true</JitOptimizationSensitive>`
 * Add NuGet/MyGet references by updating the following [test project](https://github.com/dotnet/coreclr/blob/master/tests/src/Common/test_dependencies/test_dependencies.csproj).
 * Build against the `mscorlib` facade by adding `<ReferenceLocalMscorlib>true</ReferenceLocalMscorlib>` to the test project.
 * Update relevent exclusion lists:
-  * Note that there are two build pipelines Jenkin's CI and nightly Helix - both must be updated for expected exclusion.
-    * Jenkin's CI build - see the associated `*.txt` files under `tests/` (e.g. `tests/testsUnsupportedOutsideWindows.txt`).
-    * Official nightly Helix build - see the `tests/issues.targets` file.
-  * ARM/ARM64
-    * `tests/arm/Tests.lst` and `tests/arm64/Tests.lst` are used to define the tests to run due to limitations with XUnit.
-    * These files can be manually edited or the generated using `tests/scripts/lst_creator.py`.
+  There are currently three different exclude lists. Tests.lst is currently only used by CI.
+    - `tests/issues.targets`
+    - `tests/arm/Tests.lst` - Used by Windows arm32 testing
+    - `tests/arm64/Tests.lst` - Used by Windows arm64 testing
 
 ### Creating a C# test project
 

Documentation/building/unix-test-instructions.md

@@ -81,50 +81,15 @@ sudo apt-get install libunwind8:armhf libunwind8-dev:armhf libicu-dev:armhf libl
 The following instructions assume that on the Unix machine:
 - The CoreCLR repo is cloned at `/mnt/coreclr`
 
-If DotNet is unsupported
-- The CoreFX repo is cloned at `/mnt/corefx`
-- The other platform's clone of the CoreCLR repo is mounted at `/media/coreclr`
-
-The following steps are different if DotNet is supported or not on your arch and os.
-
-### DotNet is supported
-
-build-test.sh will have setup the Core_Root directory correctly after the test build. If this was either skipped or needs to be regenerated use:
-
->`build-test.sh generatelayoutonly`
-
-To run the tests run with the --coreOverlayDir path
+build-test.sh will have setup the Core_Root directory correctly after the test build.
 
 ```bash
-~/coreclr$ tests/runtest.sh
-    --testRootDir=/mnt/coreclr/bin/tests/Linux.x64.Debug
-    --testNativeBinDir=/mnt/coreclr/bin/obj/Linux.x64.Debug/tests
-    --coreOverlayDir=/mnt/coreclr/bin/tests/Linux.x64.Debug/Tests/Core_Root
-    --copyNativeTestBin
+~/coreclr$ tests/runtest.sh x64 checked
 ```
 
-### DotNet is not supported
-
-Tests need to be built on another platform and copied over to the Unix machine for testing. Copy the test build over to the Unix machine:
-
-> `cp --recursive /media/coreclr/bin/tests/Windows_NT.x64.Debug /mnt/test/`
-
-See `runtest.sh` usage information:
-
-> `/mnt/coreclr$ tests/runtest.sh --help`
-
-Run tests (`Debug` may be replaced with `Release` or `Checked`, depending on which Configuration you've built):
+Please use the following command for help.
 
-```bash
-/mnt/coreclr$ tests/runtest.sh
-    --testRootDir=/mnt/test/Windows_NT.x64.Debug
-    --testNativeBinDir=/mnt/coreclr/bin/obj/Linux.x64.Debug/tests
-    --coreClrBinDir=/mnt/coreclr/bin/Product/Linux.x64.Debug
-    --mscorlibDir=/mnt/coreclr/bin/Product/Linux.x64.Debug
-    --coreFxBinDir=/mnt/corefx/bin/runtime/netcoreapp-Linux-Debug-x64
-```
-
-The method above will copy dependencies from the set of directories provided to create an 'overlay' directory.
+>./tests/runtest.sh -h
 
 ### Results
 
@@ -134,13 +99,21 @@ Test results will go into:
 
 ### Unsupported and temporarily disabled tests
 
-These tests are skipped by default:
-- Tests that are not supported outside Windows, are listed in:
-    > `~/coreclr/tests/testsUnsupportedOutsideWindows.txt`
-- Tests that are temporarily disabled outside Windows due to unexpected failures (pending investigation), are listed in:
-    > `~/coreclr/tests/testsFailingOutsideWindows.txt`
+Unsupported tests outside of Windows have two annotations in their csproj to
+ignore them when run.
+
+```
+<TestUnsupportedOutsideWindows>true</TestUnsupportedOutsideWindows>
+```
+
+This will write in the bash target to skip the test by returning a passing value if run outside Windows.
+
+In addition:
+```
+<DisableProjectBuild Condition="'$(TargetsUnix)' == 'true'">true</DisableProjectBuild>
+```
 
-To run only the set of temporarily disabled tests, pass in the `--runFailingTestsOnly` argument to `runtest.sh`.
+Is used to disable the build, that way if building on unix cycles are saved building/running.
 
 PAL tests
 ---------

Documentation/building/windows-test-instructions.md

@@ -21,9 +21,9 @@ This will use `crossgen.exe` to precompile the test executables before they are
 
 ## Building Other Priority Tests
 
-    C:\git\coreclr>build-test.cmd -priority=2
+    C:\git\coreclr>build-test.cmd -priority=1
 
-The number '2' is just an example. The default value (if no priority is specified) is 0. To clarify, if '2' is specified, all tests with CLRTestPriorty 0, 1 **and** 2 will be built and consequently run.
+The number '1' is just an example. The default value (if no priority is specified) is 0. To clarify, if '1' is specified, all tests with CLRTestPriorty 0 **and** 1 will be built and consequently run.
 
 ## Examples
 
@@ -35,7 +35,7 @@ To run a clean, priority 1, crossgen test pass:
 
 ### Building Individual Tests
 
-Note: buildtest.cmd or build.cmd skipnative skipmscorlib needs to be run atleast once
+Note: build-test.cmd or build.cmd skipnative needs to be run atleast once
 
 * Native Test: Build the generated Visual Studio solution or make file corresponding to Test cmake file.
 
@@ -47,7 +47,7 @@ Note: buildtest.cmd or build.cmd skipnative skipmscorlib needs to be run atleast
 
 For example to run all of the tests using your checked build:
 
-     <repo_root>\tests\runtest.cmd -checked
+     <repo_root>\tests\runtest.cmd checked
 
 This will generate a report named as `TestRun_<arch>_<flavor>.html` (e.g. `TestRun_Windows_NT__x64__Checked.html`) in the subdirectory `<repo_root>\bin\Logs`. Any tests that failed will be listed in `TestRunResults_Windows_NT__x64__Checked.err`.
 

build-test.sh

@@ -289,22 +289,18 @@ build_Tests()
     if [ $__BuildTestWrappers -ne -0 ]; then
         echo "${__MsgPrefix}Creating test wrappers..."
 
-        __XUnitWrapperBuiltMarker=${__TestBinDir}/xunit_wrapper_build
+        # Always create the test wrappers and set the exclude file.
+        export __Exclude="$__TestDir/issues.targets"
+        echo "Exclude set to $__TestDir/issues.targets"
+        build_Tests_internal "Tests_XunitWrapper" "$__ProjectDir/tests/runtest.proj" "Test Xunit Wrapper" "-BuildWrappers" "-MsBuildEventLogging= " "-TargetsWindows=false"
 
-        if [ ! -f $__XUnitWrapperBuiltMarker ]; then
-
-            build_MSBuild_projects "Tests_XunitWrapper" "$__ProjectDir/tests/runtest.proj" "Test Xunit Wrapper" "-BuildWrappers" "-MsBuildEventLogging= " "-TargetsWindows=false"
-
-            if [ $? -ne 0 ]; then
-                echo "${__MsgPrefix}Error: build failed. Refer to the build log files for details (above)"
-                exit 1
-            else
-                echo "XUnit Wrappers have been built."
-                echo "Create marker \"${__XUnitWrapperBuiltMarker}\""
-                touch $__XUnitWrapperBuiltMarker
-            fi
+        if [ $? -ne 0 ]; then
+            echo "${__MsgPrefix}Error: build failed. Refer to the build log files for details (above)"
+            exit 1
         else
-            echo "XUnit Wrappers had been built before."
+            echo "XUnit Wrappers have been built."
+            echo "Create marker \"${__XUnitWrapperBuiltMarker}\""
+            touch $__XUnitWrapperBuiltMarker
         fi
     fi
 

init-tools.cmd

@@ -47,9 +47,33 @@ echo Running %0 > "%INIT_TOOLS_LOG%"
 set /p DOTNET_VERSION=< "%~dp0DotnetCLIVersion.txt"
 if exist "%DOTNET_CMD%" goto :afterdotnetrestore
 
+REM Use x86 tools on arm64 and x86.
+REM arm32 host is not currently supported, please crossbuild.
+if /i "%PROCESSOR_ARCHITECTURE%" == "arm" (
+  echo "Error, arm32 arch not supported for build tools."
+  exit /b 1
+)
+
+if /i "%PROCESSOR_ARCHITECTURE%" == "amd64" (
+  set _Arch="x64"
+  goto ArchSet
+)
+
+REM If this is not amd64 and not arm, then we should be running on arm64 or x86
+REM either way we can (and should) use the x86 dotnet cli
+REM
+REM TODO: consume native arm64 toolset, blocked by official arm64 windows cli
+REM     : release. See https://github.com/dotnet/coreclr/issues/19614 for more
+REM     : information
+set _Arch="x86"
+
+echo "init-tools.cmd: Setting arch to %_Arch% for build tools"
+
+:ArchSet
+
 echo Installing dotnet cli...
 if NOT exist "%DOTNET_PATH%" mkdir "%DOTNET_PATH%"
-set DOTNET_ZIP_NAME=dotnet-sdk-%DOTNET_VERSION%-win-x64.zip
+set DOTNET_ZIP_NAME=dotnet-sdk-%DOTNET_VERSION%-win-%_Arch%.zip
 set DOTNET_REMOTE_PATH=https://dotnetcli.azureedge.net/dotnet/Sdk/%DOTNET_VERSION%/%DOTNET_ZIP_NAME%
 set DOTNET_LOCAL_PATH=%DOTNET_PATH%%DOTNET_ZIP_NAME%
 echo Installing '%DOTNET_REMOTE_PATH%' to '%DOTNET_LOCAL_PATH%' >> "%INIT_TOOLS_LOG%"

init-tools.sh

@@ -71,7 +71,17 @@ if [ ! -e "$__DOTNET_PATH" ]; then
             echo "Warning: build not supported on 32 bit Unix"
         fi
 
-        __PKG_ARCH=x64
+        if [ "$(uname -m)" = "armhf" ] || [ "$(uname -m)" = "arm64" ] || [ "$(uname -m)" = "aarch64" ];  then
+            if [ "$(uname -m)" = "armhf" ]; then
+                __PKG_ARCH=arm
+            fi
+
+            if [ "$(uname -m)" = "arm64" ] || [ "$(uname -m)" = "aarch64" ]; then
+                __PKG_ARCH=arm64
+            fi
+        else
+            __PKG_ARCH=x64
+        fi
 
         OSName=$(uname -s)
         case $OSName in
@@ -171,26 +181,30 @@ if [ -n "${DotNetBootstrapCliTarPath-}" ]; then
     unset ILASMCOMPILER_VERSION
 fi
 
-echo "Initializing BuildTools..."
-echo "Running: $__BUILD_TOOLS_PATH/init-tools.sh $__scriptpath $__DOTNET_CMD $__TOOLRUNTIME_DIR $__PACKAGES_DIR" >> "$__init_tools_log"
-
-# Executables restored with .NET Core 2.0 do not have executable permission flags. https://github.com/NuGet/Home/issues/4424
-chmod +x "$__BUILD_TOOLS_PATH/init-tools.sh"
-"$__BUILD_TOOLS_PATH/init-tools.sh" "$__scriptpath" "$__DOTNET_CMD" "$__TOOLRUNTIME_DIR" "$__PACKAGES_DIR" >> "$__init_tools_log"
-if [ "$?" != "0" ]; then
-    echo "ERROR: An error occurred when trying to initialize the tools." 1>&2
-    display_error_message
-    exit 1
-fi
-
-echo "Making all .sh files executable under Tools."
-# Executables restored with .NET Core 2.0 do not have executable permission flags. https://github.com/NuGet/Home/issues/4424
-ls "$__scriptpath/Tools/"*.sh | xargs chmod +x
-ls "$__scriptpath/Tools/scripts/docker/"*.sh | xargs chmod +x
+# Build tools only supported on x64
+if [ "${__PKG_ARCH}" != "x64" ] &&  [ "${__PKG_ARCH}" != "arm" ]; then
+    echo "Skipped installing build tools."
+else
+    echo "Initializing BuildTools..."
+    echo "Running: $__BUILD_TOOLS_PATH/init-tools.sh $__scriptpath $__DOTNET_CMD $__TOOLRUNTIME_DIR $__PACKAGES_DIR" >> "$__init_tools_log"
+
+    # Executables restored with .NET Core 2.0 do not have executable permission flags. https://github.com/NuGet/Home/issues/4424
+    chmod +x "$__BUILD_TOOLS_PATH/init-tools.sh"
+    "$__BUILD_TOOLS_PATH/init-tools.sh" "$__scriptpath" "$__DOTNET_CMD" "$__TOOLRUNTIME_DIR" "$__PACKAGES_DIR" >> "$__init_tools_log"
+    if [ "$?" != "0" ]; then
+        echo "ERROR: An error occurred when trying to initialize the tools." 1>&2
+        display_error_message
+        exit 1
+    fi
 
-"$__scriptpath/Tools/crossgen.sh" "$__scriptpath/Tools" $__PKG_RID
+    echo "Making all .sh files executable under Tools."
+    # Executables restored with .NET Core 2.0 do not have executable permission flags. https://github.com/NuGet/Home/issues/4424
+    ls "$__scriptpath/Tools/"*.sh | xargs chmod +x
+    ls "$__scriptpath/Tools/scripts/docker/"*.sh | xargs chmod +x
 
-mkdir -p "$(dirname "$__BUILD_TOOLS_SEMAPHORE")" && touch "$__BUILD_TOOLS_SEMAPHORE"
+    "$__scriptpath/Tools/crossgen.sh" "$__scriptpath/Tools" $__PKG_RID
 
-echo "Done initializing tools."
+    mkdir -p "$(dirname "$__BUILD_TOOLS_SEMAPHORE")" && touch "$__BUILD_TOOLS_SEMAPHORE"
 
+    echo "Done initializing tools."
+fi

tests/arm/Tests.lst

@@ -53204,14 +53204,6 @@ MaxAllowedDurationSeconds=600
 Categories=EXPECTED_PASS;Pri1
 HostStyle=0
 
-[b12011.cmd_6691]
-RelativePath=JIT\Regression\VS-ia64-JIT\V1.2-M02\b12011\b12011\b12011.cmd
-WorkingDir=JIT\Regression\VS-ia64-JIT\V1.2-M02\b12011\b12011
-Expected=0
-MaxAllowedDurationSeconds=600
-Categories=EXPECTED_PASS
-HostStyle=0
-
 [FlowControlNext.cmd_6692]
 RelativePath=CoreMangLib\cti\system\reflection\emit\flowcontrol\FlowControlNext\FlowControlNext.cmd
 WorkingDir=CoreMangLib\cti\system\reflection\emit\flowcontrol\FlowControlNext

tests/arm64/Tests.lst

@@ -70244,14 +70244,6 @@ MaxAllowedDurationSeconds=600
 Categories=JIT;EXPECTED_PASS
 HostStyle=0
 
-[b12011.cmd_9160]
-RelativePath=JIT\Regression\VS-ia64-JIT\V1.2-M02\b12011\b12011\b12011.cmd
-WorkingDir=JIT\Regression\VS-ia64-JIT\V1.2-M02\b12011\b12011
-Expected=0
-MaxAllowedDurationSeconds=600
-Categories=EXPECTED_PASS
-HostStyle=0
-
 [b14355.cmd_9161]
 RelativePath=JIT\Regression\VS-ia64-JIT\V1.2-M02\b14355\b14355\b14355.cmd
 WorkingDir=JIT\Regression\VS-ia64-JIT\V1.2-M02\b14355\b14355