Merge branch 'FXCIUnix' of https://github.com/A-And/coreclr into FXCIUnix

Author: A-And

BuildToolsVersion.txt

@@ -1 +1 @@
-2.2.0-preview1-02920-01
+2.2.0-preview1-03002-01

Documentation/building/linux-instructions.md

@@ -38,8 +38,8 @@ In order to get clang-3.9, llvm-3.9 and lldb-3.9 on Ubuntu 14.04, we need to add
     ~$ wget -O - http://llvm.org/apt/llvm-snapshot.gpg.key | sudo apt-key add -
     ~$ sudo apt-get update
 
-Note: arm clang has a known issue with CompareExchange (#15074), so for arm you have to use clang-4.0 or higher, the official build uses clang-5.0.
-    
+Note: ARM clang has a known issue with CompareExchange ([#15074](https://github.com/dotnet/coreclr/issues/15074)), so for ARM you must use clang-4.0 or higher. The official build uses clang-5.0.
+
 For other version of Debian/Ubuntu, please visit http://apt.llvm.org/.
 
 Then install the packages you need:
@@ -98,6 +98,29 @@ After the build is complete you will be able to find the output in the `bin` fol
 Build for ARM/Linux
 ===================
 
+The CI system and official builds use Docker to build ARM for Linux (for example, see the latest build [here](https://ci.dot.net/job/dotnet_coreclr/job/master/job/arm_cross_checked_ubuntu/lastSuccessfulBuild/consoleText)). The Docker container has pre-built rootfs directories containing the required tools. To build this way, do the following:
+
+* Install Docker, probably Community Edition, on Windows, Mac, or Linux, from https://www.docker.com/. Some useful post-install setup is:
+  * Linux: add your user to the docker group, this will avoid running docker with `sudo`:
+
+    `sudo usermod -a <your account name> -G docker`
+
+  * Windows: switch to Linux containers. This can be done by right clicking on the Docker icon in the lower right corner and clicking "Switch to Linux containers".
+* Build using the Docker container (this is a `bash` script, for simplicity):
+
+```
+ROOT=/Users/me/git/coreclr
+DOCKER_ARGS="run -i --rm -v ${ROOT}:/mnt/coreclr -w /mnt/coreclr -e ROOTFS_DIR=/crossrootfs/arm -e CAC_ROOTFS_DIR=/crossrootfs/x86 microsoft/dotnet-buildtools-prereqs:ubuntu-14.04-cross-e435274-20180426002420"
+docker ${DOCKER_ARGS} /mnt/coreclr/build.sh arm checked cross crosscomponent
+docker ${DOCKER_ARGS} /mnt/coreclr/build-test.sh arm checked cross generatelayoutonly
+```
+
+Make sure you update the `ROOT` environment to point to your git clone of the coreclr repo.
+
+The current Docker tag being used by the CI can be found in the `getDockerImageName` function in the [netci.groovy](https://github.com/dotnet/coreclr/blob/master/netci.groovy) file.
+
+Libunwind issue
+---------------
 Libunwind-arm requires fixes that are not included in Ubuntu 14.04, yet. The fix allows libunwind-arm not to break when it is ordered to access unaccessible memory locations.
 
 First, import the patch from the libunwind upstream: http://git.savannah.gnu.org/gitweb/?p=libunwind.git;a=commit;h=770152268807e460184b4152e23aba9c86601090
@@ -124,7 +147,7 @@ index 1ed3dbf..c643032 100644
 ```
 
 Additional optimization levels for ARM/Linux: -Oz and -Ofast
-============================================================
+------------------------------------------------------------
 
 This instruction is to enable additional optimization levels such as -Oz and -Ofast on ARM/Linux. The below table shows what we have to enable for the code optimization of the CoreCLR run-time either the size or speed on embedded devices. 
 

Documentation/coding-guidelines/clr-jit-coding-conventions.md

@@ -125,7 +125,8 @@ Note that these conventions are different from the CLR C++ Coding Conventions, d
     * [15.5.9 Global class objects](#15.5.9)
   * [15.6 Exceptions](#15.6)
   * [15.7 Code tuning for performance optimization](#15.7)
-  * [15.8 Obsoleting functions, classes and macros](#15.8)
+  * [15.8 Memory allocation](#15.8)
+  * [15.9 Obsoleting functions, classes and macros](#15.9)
 
 # <a name="4"/>4 Principles
 
@@ -1938,7 +1939,45 @@ In general, code should be written to be readable first, and optimized for perfo
 
 In the case of tight loops and code that has been analyzed to be a performance bottleneck, performance optimizations take a higher priority. Talk to the performance team if in doubt.
 
-## <a name="15.8"/>15.8 Obsoleting functions, classes and macros
+## <a name="15.8"/>15.8 Memory allocation
+
+All memory required during the compilation of a method must be allocated using the `Compiler`'s arena allocator. This allocator takes care of deallocating all the memory when compilation ends, avoiding memory leaks and simplifying memory management.
+
+However, the use of an arena allocator can increase memory usage and it's worth considering its impact when writing JIT code. Simple code changes can have a significant impact on memory usage, such as hoisting a `std::vector` variable out of a loop:
+```c++
+std::vector<int> outer; // same memory gets used for all iterations
+for (...)
+{
+    std::vector<int> inner; // this will allocate memory on every iteration
+                            // and previously allocated memory is simply wasted
+}
+```
+Node based data structures (e.g linked lists) may benefit from retaining and reusing removed nodes, provided that maintaining free lists doesn't add significant cost.
+
+The arena allocator should not be used directly. `Compiler::getAllocator(CompMemKind)` returns a `CompAllocator` object that wraps the arena allocator and supports memory usage tracking when `MEASURE_MEM_ALLOC` is enabled. It's best to use a meaningful memory kind (e.g. not `CMK_Generic`) but exceptions can be made for small allocations. `CompAllocator` objects are always pointer sized and can be freely copied and stored (useful to avoid repeated `CompMemKind` references).
+
+The `new (CompAllocator)` operator should be preferred over `CompAllocator::allocate(size_t)`. The later is intended to be used only when constructors must not be run, such as when allocating arrays for containers like `std::vector`.
+```c++
+// typical object allocation
+RangeCheck* p = new (compiler->getAllocator(CMK_RangeCheck)) RangeCheck(compiler);
+// slightly shorter alternative
+RangeCheck* p = new (compiler, CMK_RangeCheck) RangeCheck(compiler);
+// allocate an array with default initialized elements
+LclVarDsc* p = new (compiler->getAllocator(CMK_LvaTable)) LclVarDsc[lvaCount];
+// use list initialization to zero out an array
+unsigned* p = new (compiler->getAllocator(CMK_LvaTable)) unsigned[lvaTrackedCount] { };
+// use CompAllocator::allocate to allocate memory without doing any initialization...
+LclVarDsc* p = compiler->getAllocator(CMK_LvaTable).allocate<LclVarDsc>(lvaCount);
+// ... and construct elements in place as needed
+new (&p[i], jitstd::placement_t()) LclVarDsc(compiler)
+```
+Note that certain classes (e.g. `GenTree`) provide their own `new` operator overloads, those should be used instead of the general purpose `new (CompAllocator)` operator.
+
+`jitstd` container classes accept a `CompAllocator` object by implicit conversion from `CompAllocator` to `jitstd::allocator`.
+
+Debug/checked code that needs to allocate memory outside of method compilation can use the `HostAllocator` class and the associated `new` operator. This is a normal memory allocator that requires manual memory deallocation.
+
+## <a name="15.9"/>15.9 Obsoleting functions, classes and macros
 
 The Visual C++ compiler has support built in for marking various user defined constructs as deprecated. This functionality is accessed via one of two mechanisms:
 

ILAsmVersion.txt

@@ -1 +1 @@
-2.2.0-preview1-26621-01
+3.0.0-preview1-26703-04

build-test.cmd

@@ -358,6 +358,19 @@ REM ===
 REM === Create the test overlay
 REM ===
 REM =========================================================================================
+
+set RuntimeIdArg=
+if defined __RuntimeId (
+    set RuntimeIdArg=-RuntimeID="%__RuntimeId%"
+)
+
+set TargetsWindowsArg=
+if "%__TargetsWindows%"=="1" (
+    set TargetsWindowsArg=-TargetsWindows=true
+) else if "%__TargetsWindows%"=="0" (
+    set TargetsWindowsArg=-TargetsWindows=false
+)
+
 echo %__MsgPrefix%Creating test overlay...
 
 set __BuildLogRootName=Tests_Overlay_Managed
@@ -403,19 +416,6 @@ if defined __SkipManaged goto SkipBuildingWrappers
 
 echo %__MsgPrefix%Creating test wrappers...
 
-set RuntimeIdArg=
-set TargetsWindowsArg=
-
-if defined __RuntimeId (
-    set RuntimeIdArg=-RuntimeID="%__RuntimeId%"
-)
-
-if "%__TargetsWindows%"=="1" (
-    set TargetsWindowsArg=-TargetsWindows=true
-) else if "%__TargetsWindows%"=="0" (
-    set TargetsWindowsArg=-TargetsWindows=false
-)
-
 set __BuildLogRootName=Tests_XunitWrapper
 set __BuildLog=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.log
 set __BuildWrn=%__LogsDir%\%__BuildLogRootName%_%__BuildOS%__%__BuildArch%__%__BuildType%.wrn

build-test.sh

@@ -193,25 +193,18 @@ generate_layout()
 
 generate_testhost()
 {
-    generate_corefx_helpers
     export TEST_HOST=$xUnitTestBinBase/testhost
 
     if [ -d "${TEST_HOST}" ]; then
         rm -rf $TEST_HOST
     fi
-    
+
+    echo "${__MsgPrefix}Creating test overlay..."    
     mkdir -p $TEST_HOST
 
     build_Tests_internal "Tests_Generate_TestHost" "${__ProjectDir}/tests/runtest.proj" "-testHost" "Creating test host"
 }
 
-generate_corefx_helpers()
-{
-    local coreFXTestSetupUtilityName=CoreFX.TestUtils.TestFileSetup
-    local coreFXTestSetupUtility="${__ProjectDir}/tests/src/Common/CoreFX/TestFileSetup/${coreFXTestSetupUtilityName}.csproj"
-    
-    build_Tests_internal "${coreFXTestSetupUtilityName}" "${coreFXTestSetupUtility}" " " "Building CoreFX Test Helper Projects"
-}
 
 build_Tests()
 {
@@ -767,22 +760,21 @@ fi
 # Specify path to be set for CMAKE_INSTALL_PREFIX.
 # This is where all built CoreClr libraries will copied to.
 export __CMakeBinDir="$__BinDir"
-echo check for $__BinDir
-if [ ! -d "$__BinDir" ]; then
+if [ [ ! -d "$__BinDir" ] || [ ! -d "$__BinDir/bin" ] ]; then
+    if [ [ -z "$__GenerateLayoutOnly" ] && -z [ "$__GenerateTestHostOnly" ] ]; then
 
-    echo "Cannot find build directory for the CoreCLR Product."
-    echo "Please make sure CoreCLR and native tests are built before building managed tests."
-    echo "Example use: './build.sh $__BuildArch $__BuildType' "
+        echo "Cannot find build directory for the CoreCLR native tests."
+        echo "Please make sure native tests are built before building managed tests."
+        echo "Example use: './build.sh $__BuildArch $__BuildType' without -skiptests switch"
+    else
+        echo "Cannot find build directory for the CoreCLR Product."
+        echo "Please make sure CoreCLR and native tests are built before building managed tests."
+        echo "Example use: './build.sh $__BuildArch $__BuildType' "
+        fi
     exit 1
 fi
 
-if [[ (-z "$__GenerateLayoutOnly") && (-z "$__GenerateTestHostOnly") && (! -d "$__BinDir/bin") ]]; then
-    
-    echo "Cannot find build directory for the CoreCLR native tests."
-    echo "Please make sure native tests are built before building managed tests."
-    echo "Example use: './build.sh $__BuildArch $__BuildType' without -skiptests switch"
-    exit 1
-fi
+
 
 # Configure environment if we are doing a cross compile.
 if [ $__CrossBuild == 1 ]; then

clrdefinitions.cmake

@@ -195,6 +195,9 @@ add_definitions(-DFEATURE_STRONGNAME_MIGRATION)
 if (CLR_CMAKE_PLATFORM_UNIX OR CLR_CMAKE_TARGET_ARCH_ARM64)
     add_definitions(-DFEATURE_STUBS_AS_IL)
 endif ()
+if (FEATURE_NGEN_RELOCS_OPTIMIZATIONS)
+   add_definitions(-DFEATURE_NGEN_RELOCS_OPTIMIZATIONS)
+endif(FEATURE_NGEN_RELOCS_OPTIMIZATIONS)
 add_definitions(-DFEATURE_SVR_GC)
 add_definitions(-DFEATURE_SYMDIFF)
 add_definitions(-DFEATURE_TIERED_COMPILATION)

cross/armel/tizen-fetch.sh

@@ -51,7 +51,7 @@ if [ ! -d $TMPDIR ]; then
 	mkdir -p $TMPDIR 
 fi
 
-TIZEN_URL=http://download.tizen.org/snapshots/tizen
+TIZEN_URL=http://download.tizen.org/releases/milestone/tizen
 BUILD_XML=build.xml
 REPOMD_XML=repomd.xml
 PRIMARY_XML=primary.xml
@@ -154,8 +154,8 @@ fetch_tizen_pkgs()
 	done
 }
 
-Inform "Initialize arm 4.0-base"
-fetch_tizen_pkgs_init arm 4.0-base
+Inform "Initialize arm base"
+fetch_tizen_pkgs_init standard base
 Inform "fetch common packages"
 fetch_tizen_pkgs armv7l gcc glibc glibc-devel libicu libicu-devel
 fetch_tizen_pkgs noarch linux-glibc-devel
@@ -164,8 +164,8 @@ fetch_tizen_pkgs armv7l lldb lldb-devel libgcc libstdc++ libstdc++-devel libunwi
 Inform "fetch corefx packages"
 fetch_tizen_pkgs armv7l libcom_err libcom_err-devel zlib zlib-devel libopenssl libopenssl-devel krb5 krb5-devel libcurl libcurl-devel
 
-Inform "Initialize standard 4.0-unified"
-fetch_tizen_pkgs_init standard 4.0-unified
+Inform "Initialize standard unified"
+fetch_tizen_pkgs_init standard unified
 Inform "fetch corefx packages"
 fetch_tizen_pkgs armv7l gssdp gssdp-devel
 

cross/armel/tizen/tizen-dotnet.ks

@@ -8,8 +8,8 @@ rootpw tizen
 desktop --autologinuser=root
 user --name root  --groups audio,video --password 'tizen'
 
-repo --name=standard  --baseurl=http://download.tizen.org/snapshots/tizen/4.0-unified/latest/repos/standard/packages/ --ssl_verify=no
-repo --name=base      --baseurl=http://download.tizen.org/snapshots/tizen/4.0-base/latest/repos/arm/packages/ --ssl_verify=no
+repo --name=standard  --baseurl=http://download.tizen.org/releases/milestone/tizen/unified/latest/repos/standard/packages/ --ssl_verify=no
+repo --name=base      --baseurl=http://download.tizen.org/releases/milestone/tizen/base/latest/repos/standard/packages/ --ssl_verify=no
 
 %packages
 tar

dependencies.props

@@ -23,18 +23,18 @@
 
   <!-- Source of truth for dependency tooling: the commit hash of the dotnet/versions master branch as of the last auto-upgrade. -->
   <PropertyGroup>
-    <CoreFxCurrentRef>232e7f9230fcc7c6c8ea8c955e113f897403a0e7</CoreFxCurrentRef>
-    <CoreClrCurrentRef>232e7f9230fcc7c6c8ea8c955e113f897403a0e7</CoreClrCurrentRef>
-    <BuildToolsCurrentRef>3b09a6a5fd370dfb8b7af43ca0a93114efdaa0a1</BuildToolsCurrentRef>
-    <PgoDataCurrentRef>1f57485574742714a2401175bce892597bbf86bb</PgoDataCurrentRef>
+    <CoreFxCurrentRef>70c44f38436f4ae4734e792e06cefba1e412f9fc</CoreFxCurrentRef>
+    <CoreClrCurrentRef>d8a64e527e67cae4bb3c9d93379fd43aa317165b</CoreClrCurrentRef>
+    <BuildToolsCurrentRef>3b2844e866d3898d43d98aad0e3e57dd5675d259</BuildToolsCurrentRef>
+    <PgoDataCurrentRef>d8a64e527e67cae4bb3c9d93379fd43aa317165b</PgoDataCurrentRef>
   </PropertyGroup>
 
   <!-- Tests/infrastructure dependency versions. -->
   <PropertyGroup>
-    <MicrosoftPrivateCoreFxNETCoreAppPackageVersion>4.6.0-preview1-26621-01</MicrosoftPrivateCoreFxNETCoreAppPackageVersion>
-    <MicrosoftNETCorePlatformsPackageVersion>2.2.0-preview1-26621-01</MicrosoftNETCorePlatformsPackageVersion>
-    <PgoDataPackageVersion>99.99.99-master-20180621-0050</PgoDataPackageVersion>
-    <MicrosoftNETCoreRuntimeCoreCLRPackageVersion>2.2.0-preview1-26621-01</MicrosoftNETCoreRuntimeCoreCLRPackageVersion>
+    <MicrosoftPrivateCoreFxNETCoreAppPackageVersion>4.6.0-preview1-26628-03</MicrosoftPrivateCoreFxNETCoreAppPackageVersion>
+    <MicrosoftNETCorePlatformsPackageVersion>3.0.0-preview1-26628-03</MicrosoftNETCorePlatformsPackageVersion>
+    <PgoDataPackageVersion>99.99.99-master-20180703-0030</PgoDataPackageVersion>
+    <MicrosoftNETCoreRuntimeCoreCLRPackageVersion>3.0.0-preview1-26703-04</MicrosoftNETCoreRuntimeCoreCLRPackageVersion>
     <XunitPackageVersion>2.2.0-beta2-build3300</XunitPackageVersion>
     <XunitConsoleNetcorePackageVersion>2.2.0-preview1-02830-02</XunitConsoleNetcorePackageVersion>
     <XunitPerformanceApiPackageVersion>1.0.0-beta-build0015</XunitPerformanceApiPackageVersion>
@@ -51,7 +51,7 @@
   <!-- Package versions used as toolsets -->
   <PropertyGroup>
     <FeedTasksPackage>Microsoft.DotNet.Build.Tasks.Feed</FeedTasksPackage>
-    <FeedTasksPackageVersion>2.2.0-preview1-02920-01</FeedTasksPackageVersion>
+    <FeedTasksPackageVersion>2.2.0-preview1-03002-01</FeedTasksPackageVersion>
   </PropertyGroup>
 
   <!-- Publish symbol build task package -->