CoreFX CI Unix

Documentation/building/testing-with-corefx.md

@@ -29,42 +29,76 @@ Use the following instructions to test a change to the dotnet/coreclr repo using
 [run-corefx-tests.py](https://github.com/dotnet/coreclr/blob/master/tests/scripts/run-corefx-tests.py) will clone dotnet/corefx and run steps 2-4 above automatically.  It is primarily intended to be run by the dotnet/coreclr CI system, but it might provide a useful reference or shortcut for individuals running the tests locally.
 
 ## Using the built CoreCLR testhost 
-**These instructions are currently Windows only.**
 
 Instead of copying CoreCLR binaries you can also test your changes with an existing CoreFX build or CoreCLR's CI assemblies
 
 ### Locally-built CoreFX 
+
 Once you have finished steps 1, 2. and 4. above execute the following instructions to test your local CLR changes with the built-CoreFX changes.
 
-1. From `<coreclr_root>` run `build-test.cmd <arch> <build_type> skipmanaged` to generate the test host.
+1. From `<coreclr_root>` run
+`build-test.cmd <arch> <build_type> buildtesthostonly` 
+
+-or-
+
+`build-test.sh <arch> <build_type> generatetesthostonly` 
+
+to generate the test host.
 2. Navigate to `<corefx_root>\bin\tests\` and then the test you would like to run
 3. Run
 
 ```cmd
 <coreclr_root>\bin\<os>.<arch>.<build_type>\testhost\dotnet.exe <corefx_root>\bin\tests\<testname>\xunit.console.netcore.exe <testname>.dll
 ```
+-or-
+
+```sh
+<coreclr_root>/bin/<os>.<arch>.<build_type>/testhost/dotnet <corefx_root>/bin/tests/<testname>/xunit.console.netcore.exe <testname>.dll
+```
+
 followed by any extra command-line arguments.
 
 For example to run .NET Core Windows tests from System.Collections.Tests with an x64 Release build of CoreCLR.
 ```
-pushd C:\corefx\bin\tests\System.Collections.Tests
+cd C:\corefx\bin\tests\System.Collections.Tests
 C:\coreclr\bin\tests\Windows_NT.x64.Release\testhost\dotnet.exe .\xunit.console.netcore.exe .\System.Collections.Tests.dll -notrait category=nonnetcoretests -notrait category=nonwindowstests
 ```
 
+-or-
+
+```
+cd ~/corefx/bin/tests/System.Collections.Tests
+~/coreclr/bin/tests/Linux.x64.Release/testhost/dotnet .\xunit.console.netcore.exe .\System.Collections.Tests.dll -notrait category=nonnetcoretests -notrait category=nonlinuxtests
+```
+
 ### CI Script
-CoreCLR has an alternative way to run CoreFX tests, built for PR CI jobs. To run tests against pre-built binaries you can execute the following from the CoreCLR repo root:
+CoreCLR has an alternative way to run CoreFX tests, built for PR CI jobs. 
 
-1. `.\build.cmd <arch> <build_type>`
-2. `.\build-test.cmd <arch> <build_type> skipmanaged` - generates the test host
+To run tests against pre-built binaries you can execute the following from the CoreCLR repo root:
+
+#### Windows
+1. `.\build.cmd <arch> <build_type> skiptests`
+2. `.\build-test.cmd <arch> <build_type> buildtesthostonly` - generates the test host
 3. `.\tests\runtest.cmd <arch> <build_type> corefxtests|corefxtestsall` - runs CoreFX tests
 
+#### Linux and OSX
+1. `./build.sh <arch> <build_type> skiptests`
+2. `./build-test.sh <arch> <build_type>  generatetesthostonly`
+3. `./tests/runtest.sh  --corefxtests|--corefxtestsall --testHostDir=<path_to_testhost> --coreclr-src<path_to_coreclr>`
+
 CoreFXTests - runs all tests defined in TopN.Windows.CoreFX.issues.json or the test list specified with the argument `CoreFXTestList`
 CoreFXTestsAll - runs all tests available in the test list found at the URL in `.\coreclr\tests\CoreFX\CoreFXTestListURL.txt`.
+#### Linux - specific
+&lt;path_to_testhost&gt; - path to the coreclr test host built in step 2.
+&lt;path_to_coreclr&gt; - path to coreclr source 
 
 ### Helix Testing
 To use Helix-built binaries, substitute the URL in `.\coreclr\tests\CoreFX\CoreFXTestListURL.txt` with one acquired from a Helix test run and run the commands above.
 
-### Test List Format
+#### Workflow
+The CoreFX tests CI jobs run against cached test binaries in blob storage. This means that tests might need to be disabled until the test binaries are refreshed as breaking changes are merged in both CoreCLR and CoreFX. If you suspect a test is not failing because of a functional regression, but rather because it's stale you can add it to either the [Windows](https://github.com/dotnet/coreclr/blob/master/tests/CoreFX/TopN.CoreFX.x64.Windows.issues.json) or [Unix](https://github.com/dotnet/coreclr/blob/master/tests/CoreFX/TopN.CoreFX.x64.Unix.issues.json) test exclusion lists.
+
+#### Test List Format
 The tests defined in TopN.Windows.CoreFX.issues.json or the test list specified with the argument `CoreFXTestList` should conform to the following format -
 ```json
     {

build-test.sh

@@ -187,8 +187,24 @@ generate_layout()
     if [ ! -f "${__BuildToolsDir}/Microsoft.CSharp.targets" ]; then
         ln -s "${__BuildToolsDir}/Microsoft.CSharp.Targets" "${__BuildToolsDir}/Microsoft.CSharp.targets"
     fi
+
 }
 
+generate_testhost()
+{
+    export TEST_HOST=$xUnitTestBinBase/testhost
+
+    if [ -d "${TEST_HOST}" ]; then
+        rm -rf $TEST_HOST
+    fi
+
+    echo "${__MsgPrefix}Creating test overlay..."    
+    mkdir -p $TEST_HOST
+
+    build_Tests_internal "Tests_Generate_TestHost" "${__ProjectDir}/tests/runtest.proj" "-testHost" "Creating test host"
+}
+
+
 build_Tests()
 {
     __TestDir=$__ProjectDir/tests
@@ -510,6 +526,7 @@ __RunTests=0
 __RebuildTests=0
 __BuildTestWrappers=0
 __GenerateLayoutOnly=
+__GenerateTestHostOnly=
 __priority1=
 CORE_ROOT=
 
@@ -641,11 +658,9 @@ while :; do
         generatelayoutonly)
             __GenerateLayoutOnly=1
             ;;
-
         generatetesthostonly)
-            exit 0
+            __GenerateTestHostOnly=1
             ;;
-
         buildagainstpackages)
             __BuildAgainstPackagesArg=1
             ;;
@@ -745,15 +760,22 @@ fi
 # Specify path to be set for CMAKE_INSTALL_PREFIX.
 # This is where all built CoreClr libraries will copied to.
 export __CMakeBinDir="$__BinDir"
+if [ [ ! -d "$__BinDir" ] || [ ! -d "$__BinDir/bin" ] ]; then
+    if [ [ -z "$__GenerateLayoutOnly" ] && [ -z "$__GenerateTestHostOnly" ] ]; then
 
-if [ ! -d "$__BinDir" ] || [ ! -d "$__BinDir/bin" ]; then
-
-    echo "Cannot find build directory for the CoreCLR Product or native tests."
-    echo "Please make sure CoreCLR and native tests are built before building managed tests."
-    echo "Example use: './build.sh $__BuildArch $__BuildType' without -skiptests switch"
+        echo "Cannot find build directory for the CoreCLR native tests."
+        echo "Please make sure native tests are built before building managed tests."
+        echo "Example use: './build.sh $__BuildArch $__BuildType' without -skiptests switch"
+    else
+        echo "Cannot find build directory for the CoreCLR Product."
+        echo "Please make sure CoreCLR and native tests are built before building managed tests."
+        echo "Example use: './build.sh $__BuildArch $__BuildType' "
+        fi
     exit 1
 fi
 
+
+
 # Configure environment if we are doing a cross compile.
 if [ $__CrossBuild == 1 ]; then
     export CROSSCOMPILE=1
@@ -770,12 +792,17 @@ initTargetDistroRid
 __CoreClrVersion=1.1.0
 __sharedFxDir=$__BuildToolsDir/dotnetcli/shared/Microsoft.NETCore.App/$__CoreClrVersion/
 
-echo "Building Tests..."
 
-if [ -z "$__GenerateLayoutOnly" ]; then
+if [[ (-z "$__GenerateLayoutOnly") && (-z "$__GenerateTestHostOnly") ]]; then
+    echo "Building Tests..."
     build_Tests
 else
+    echo "Generating test layout..."
     generate_layout
+    if [ ! -z "$__GenerateTestHostOnly" ]; then
+        echo "Generating test host..."        
+        generate_testhost
+    fi
 fi
 
 if [ $? -ne 0 ]; then

tests/CoreFX/CoreFXTestListURL_Linux.txt

@@ -1 +1 @@
-https://cloudcijobs.blob.core.windows.net/corertci/TestList_Linux.json
+https://cloudcijobs.blob.core.windows.net/coreclrci/CoreFXArchives_Linux/TestList.json

tests/CoreFX/CoreFXTestListURL_OSX.txt

@@ -1 +1 @@
-https://cloudcijobs.blob.core.windows.net/corertci/TestList_OSX.json
+https://cloudcijobs.blob.core.windows.net/coreclrci/CoreFXArchives_OSX/TestList.json