Merge

Author: CarolEidt

.gitignore

@@ -152,6 +152,7 @@ publish/
 *.nuget.g.targets
 *.nuget.cache
 **/packages/*
+*.nuget.dgspec.json
 project.lock.json
 project.assets.json
 

Documentation/botr/exceptions.md

@@ -301,5 +301,3 @@ Miscellaneous
 =============
 
 There are actually a lot of macros involved in EX_TRY. Most of them should never, ever, be used outside of the macro implementations.
-
-One set, BEGIN_EXCEPTION_GLUE / END_EXCEPTION_GLUE, deserves special mention. These were intended to be transitional macros, and were to be replaced with more appropriate macros in the Whidbey project. Of course, they worked just fine, and so they weren't all replaced. Ideally, all instances will be converted during a "cleanup" milestone, and the macros removed. In the meantime, any CLR dev tempted to use them should resist, and instead write EX_TRY/EX_CATCH/EX_CATCH_END or EX_CATCH_HRESULT.

Documentation/botr/ryujit-overview.md

@@ -195,7 +195,9 @@ At this point, a number of analyses and transformations are done on the flowgrap
 
 At this point, a number of properties are computed on the IR, and must remain valid for the remaining phases. We will call this “normalization”
 
-* `lvaMarkLocalVars` – set the reference counts (raw and weighted) for lclVars, sort them, and determine which will be tracked (currently up to 128). Note that after this point any transformation that adds or removes lclVar references must update the reference counts.
+* `lvaMarkLocalVars` – if this jit is optimizing, set the reference counts (raw and weighted) for lclVars, sort them, and determine which
+will be tracked (currently up to 512). If not optimizing, all locals are given an implicit reference count of one. Reference counts are not incrementally
+maintained. They can be recomputed if accurate counts are needed.
 * `optOptimizeBools` – this optimizes Boolean expressions, and may change the flowgraph (why is it not done prior to reachability and dominators?)
 * Link the trees in evaluation order (setting `gtNext` and `gtPrev` fields): and `fgFindOperOrder()` and `fgSetBlockOrder()`.
 
@@ -492,7 +494,16 @@ TreeNodeInfo:
 
 ## LclVar phase-dependent properties
 
-Prior to normalization, the reference counts (`lvRefCnt` and `lvRefCntWtd`) are not valid. After normalization they must be updated when lclVar references are added or removed.
+LclVar ref counts track the number of uses and weighted used of a local in the jit IR. There are two sequences of phases over which ref counts are valid,
+tracked via `lvaRefCountState`: an early sequence (state `RCS_EARLY`) and the normal sequence (state `RCS_NORMAL`). Requests for ref counts via `lvRefCnt`
+and `lvRefCntWtd` must be aware of the ref count state.
+
+Before struct promotion the ref counts are invalid. Struct promotion enables `RCS_EARLY` and it and subsequent phases through morph compute and uses ref counts
+on some locals to guide some struct optimizations. After morph the counts go back to longer being valid.
+
+The `RCS_NORMAL` sequence begins at normalization. Ref counts are computed and generally available via for the rest of the compilation phases.
+The counts are not incrementally maintained and may go stale as the IR is optimized or transformed, or maybe very approximate if the jit is not optimizing.
+They can be recomputed via `lvaComputeRefCounts` at points where accurate counts are valuable. Currently this happens before and after lower.
 
 # Supporting technologies and components
 

Documentation/building/unix-test-instructions.md

@@ -24,17 +24,13 @@ Please note that this builds the Priority 0 tests. To build priority 1:
 
 During development there are many instances where building an individual test is fast and necessary. All of the necessary tools to build are under `coreclr/Tools`. It is possible to use `coreclr/Tools/MSBuild.dll` as you would normally use MSBuild with a few caveats.
 
-Note that `coreclr/Tools/msbuild.sh` exists as well to make the call shorter.
+Note that `coreclr/dotnet.sh` exists as well to make the call shorter.
 
 **!! Note !! -- Passing /p:__BuildOs=[OSX|Linux] is required.** 
 
 ## Building an Individual Test Example
 
->`coreclr/Tools/msbuild.sh /maxcpucount  coreclr/tests/src/JIT/CodeGenBringUpTests/Array1.csproj /p:__BuildType=Release /p:__BuildOS=OSX`
-
-Or
-
->`coreclr/Tools/dotnetcli/dotnet coreclr/Tools/MSBuild.dll /maxcpucount coreclr/tests/src/JIT/CodeGenBringUpTests/Array1.csproj /p:__BuildType=Release /p:__BuildOS=OSX`
+>`coreclr/dotnet.sh msbuild /maxcpucount  coreclr/tests/src/JIT/CodeGenBringUpTests/Array1.csproj /p:__BuildType=Release /p:__BuildOS=OSX`
 
 ## Aarch64/armhf multi-arch
 

Documentation/design-docs/event-counter.md

@@ -35,23 +35,28 @@ We believe adding some new top-level types will satisfy these requests:
         EventCounter(string name, EventSource eventSource);
         string DisplayName;
         void WriteMetric(float metric);
-        
+        void AddMetaData(string key, string value);
     }
 
     class PollingCounter {
         PollingCounter(string name, EventSource eventSource Func<float> getMetricFunction);
         string DisplayName;
+        void AddMetaData(string key, string value);
     }
 
     class IncrementingEventCounter {
         IncrementingEventCounter(string name, EventSource eventSource);
         string DisplayName;
-        Increment(float increment = 1);
+        TimeSpan DisplayRateTimeScale;
+        void Increment(float increment = 1);
+        void AddMetaData(string key, string value);
     }
 
     class IncrementingPollingCounter {
         IncrementingPollingCounter(string name, EventSource eventSource, Func<float> getCountFunction);
         string DisplayName;
+        TimeSpan DisplayRateTimeScale;
+        void AddMetaData(string key, string value);
     }
 
 
@@ -60,16 +65,18 @@ EventCounter does what it has always done, producing a set of 5 stats (Min/Max/M
 On the wire EventCounter and PollingCounter both produce an event with name "EventCounters" and example body:
 
     Payload = {
-        DisplayName: "Request Bytes"
+        DisplayName: "Request Bytes",
+        DisplayRateTimeScale: "1",
         Name: "request-bytes",
         Mean: 12.32,
-        StandardDeviation: 2.45
-        Count: 7
-        Min: -3.4
-        Max: 22.98
-        IntervalSec: 1.00324
-        Series: "Interval=1"
-        CounterType: "Mean"
+        StandardDeviation: 2.45,
+        Count: 7,
+        Min: -3.4,
+        Max: 22.98,
+        IntervalSec: 1.00324,
+        Series: "Interval=1",
+        CounterType: "Mean",
+        MetaData: "key1=value1,key2=value2,key3=value3"
     }
 
 
@@ -79,12 +86,14 @@ IncrementingEventCounter and IncrementingPollingCounter, unlike the previous two
 On the wire IncrementingEventCounter and IncrementingPollingCounter both produce an event with the name "EventCounters" and example body:
 
     Payload = {
-        DisplayName: "Exceptions Thrown"
-        Name: "exceptions-thrown"
-        Increment: 246
-        IntervalSec: 1.0043
-        Series: "Interval=1"
-        CounterType: "Sum"
+        DisplayName: "Exceptions Thrown",
+        DisplayRateTimeScale: "1",
+        Name: "exceptions-thrown",
+        Increment: 246,
+        IntervalSec: 1.0043,
+        Series: "Interval=1",
+        CounterType: "Sum",
+        MetaData: "key1=value1,key2=value2,key3=value3"
     }
 
 
@@ -99,4 +108,4 @@ For EventCounter and PollingCounter we expect simple viewers to use the display
 
 ### Metadata
 
-I added fields for DisplayName and CounterType directly to the payload. Vance suggested adding a 'Metadata' field, which I would be OK with, but I see no benefit to using it for fields we can anticipate will exist in advance, and we already have strongly typed APIs that generate their values. In the future if we added an API that let EventCounter consumers set arbitrary key value pairs on the counter, that seems like the data we'd want to encode in a Metadata string.
+To add any optional metadata about the counters that we do not already provide a way of encoding, users can call the `AddMetaData(string key, string value)` API. This API exists on all variants of the Counter APIs, and allows users to add one or many key-value pairs of metadata, which is dumped to the Payload as a comma-separated string value. This API exists so that users can add any metadata about their Counter that is not known to us and is different from the ones we provide by default (i.e. `DisplayName`, `CounterType`, `DisplayRateTimeScale`).

Documentation/project-docs/linux-performance-tracing.md

@@ -92,7 +92,7 @@ need to fetch EXACTLY the right version of crossgen for the runtime you happen t
 using.  If you place the crossgen tool in the same directory as the .NET Runtime DLLs (e.g. libcoreclr.so), then perfcollect can
 find it and add the framework symbols to the trace file for you.
 
-Normally when you create a .NET application it just generates the DLL for the code you wrote, using a shared copy of the runtime
+Normally when you create a .NET application, it just generates the DLL for the code you wrote, using a shared copy of the runtime
 for the rest.   However you can also generate what is called a 'self-contained' version of an application and this contains all 
 runtime DLLs.  It turns out that the crossgen tool is part of the Nuget package that is used to create these self-contained apps, so
 one way of getting the right crossgen tool is to create a self contained package of any application.   
@@ -104,8 +104,8 @@ So you could do the following
    > dotnet new console
    > dotnet publish --self-contained -r linux-x64
    >```
-Which creates a new helloWorld application and builds it as a self-contained app.    The only subtlty here is that if you have
-multiple versions of the .NET Runtime installed the instructcions above will use the latest.  As long as your app also uses
+Which creates a new helloWorld application and builds it as a self-contained app.    The only subtlety here is that if you have
+multiple versions of the .NET Runtime installed, the instructions above will use the latest.  As long as your app also uses
 the latest (likely) then these instructions will work without modification.   
 
 As a side effect of creating the self-contained application the dotnet tool will download a nuget package 
@@ -127,11 +127,11 @@ issue should go away.   This only has to be one once per machine (until you upda
 
 ### Alternative: Turn off use of precompiled code
 
-If you don't have the abiltiy to update the .NET Rutnime (to add crossgen), or if the above procedure did not work
-for some reasion, there is another approach to getting framework symbols.   You can tell the runtime to simply 
-not use the precompiled framework code.   The all code will be Just in time compiled and the special crossgen tool
+If you don't have the abiltiy to update the .NET Runtime (to add crossgen), or if the above procedure did not work
+for some reason, there is another approach to getting framework symbols.   You can tell the runtime to simply 
+not use the precompiled framework code.   The code will be Just in time compiled and the special crossgen tool
 is not needed.   This works, but will increase startup time for your code by something like a second or two.  If you 
-can tolerate that (you probably can), then this is an alternative.   You were already setting envinronment variables
+can tolerate that (you probably can), then this is an alternative.   You were already setting environment variables
 in order to get symbols, you simply need to add one more.
 	> ```bash 
 	> export COMPlus_ZapDisable=1

Documentation/workflow/RunningTests.md

@@ -31,12 +31,10 @@ Note: The CoreCLR must be built prior to building an individual test. See first
 
 ### Examples
 
-* Using the `msbuild.sh` script
-  * `coreclr/Tools/msbuild.sh /maxcpucount coreclr/tests/src/JIT/CodeGenBringUpTests/Array1.csproj /p:__BuildType=Release /p:__BuildOS=OSX`
-* Calling `MSBuild.dll` directly
-  * `coreclr/Tools/dotnetcli/dotnet coreclr/Tools/MSBuild.dll /maxcpucount coreclr/tests/src/JIT/CodeGenBringUpTests/Array1.csproj /p:__BuildType=Release /p:__BuildOS=OSX`
+* Using the `dotnet.sh` script
+  * `coreclr/dotnet.sh msbuild /maxcpucount coreclr/tests/src/JIT/CodeGenBringUpTests/Array1.csproj /p:__BuildType=Release /p:__BuildOS=OSX`
 
 ## Additional Documents
 
 * [Windows](https://github.com/dotnet/coreclr/blob/master/Documentation/building/windows-test-instructions.md)
-* [Non-Windows](https://github.com/dotnet/coreclr/blob/master/Documentation/building/unix-test-instructions.md)
+* [Non-Windows](https://github.com/dotnet/coreclr/blob/master/Documentation/building/unix-test-instructions.md)

azure-pipelines.yml

@@ -9,22 +9,22 @@ variables:
 resources:
   containers:
   - container: ubuntu_1404_arm_cross_build_image
-    image: microsoft/dotnet-buildtools-prereqs:ubuntu-14.04-cross-e435274-20180426002420
+    image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-14.04-cross-e435274-20180426002420
 
   - container: ubuntu_1604_arm64_cross_build_image
-    image: microsoft/dotnet-buildtools-prereqs:ubuntu-16.04-cross-arm64-a3ae44b-20180315221921
+    image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-arm64-a3ae44b-20180315221921
 
   - container: musl_x64_build_image
-    image: microsoft/dotnet-buildtools-prereqs:alpine-3.6-WithNode-f4d3fe3-20181220200247
+    image: mcr.microsoft.com/dotnet-buildtools/prereqs:alpine-3.6-WithNode-f4d3fe3-20181220200247
 
   - container: musl_arm64_build_image
-    image: microsoft/dotnet-buildtools-prereqs:ubuntu-16.04-cross-arm64-alpine10fcdcf-20190208200917
+    image: mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-16.04-cross-arm64-alpine10fcdcf-20190208200917
 
   - container: centos7_x64_build_image
-    image: microsoft/dotnet-buildtools-prereqs:centos-7-d485f41-20173404063424
+    image: mcr.microsoft.com/dotnet-buildtools/prereqs:centos-7-d485f41-20173404063424
 
   - container: centos6_x64_build_image
-    image: microsoft/dotnet-buildtools-prereqs:centos-6-376e1a3-20174311014331
+    image: mcr.microsoft.com/dotnet-buildtools/prereqs:centos-6-376e1a3-20174311014331
 
 trigger:
 - master
@@ -149,6 +149,12 @@ jobs:
           testGroup: outerloop
         ${{ if eq(variables['Build.DefinitionName'], 'coreclr-outerloop-jitminopts-jitstress1-jitstress2') }}:
           testGroup: outerloop-jitminopts-jitstress1-jitstress2
+        ${{ if eq(variables['Build.DefinitionName'], 'coreclr-outerloop-jitstressregs') }}:
+          testGroup: outerloop-jitstressregs
+        ${{ if eq(variables['Build.DefinitionName'], 'coreclr-outerloop-jitstress2-jitstressregs') }}:
+          testGroup: outerloop-jitstress2-jitstressregs
+        ${{ if eq(variables['Build.DefinitionName'], 'coreclr-outerloop-gcstress0x3-gcstress0xc') }}:
+          testGroup: outerloop-gcstress0x3-gcstress0xc
 
 # CI
 - ${{ if and(eq(variables['System.TeamProject'], 'public'), in(variables['Build.Reason'], 'IndividualCI', 'BatchedCI')) }}:

build-packages.cmd

@@ -33,6 +33,11 @@ if [!processedArgs!]==[] (
 
 :ArgsDone
 
+call "%__ProjectDir%"\setup_vs_tools.cmd
+
+REM setup_vs_tools.cmd will correctly echo error message.
+if NOT '%ERRORLEVEL%' == '0' exit /b 1
+
 call %__ProjectDir%/dotnet.cmd msbuild /nologo /verbosity:minimal /clp:Summary /nodeReuse:false^
   /p:__BuildOS=Windows_NT /flp:v=detailed;Append;LogFile=build-packages.log^
   /l:BinClashLogger,Tools/Microsoft.DotNet.Build.Tasks.dll;LogFile=binclash.log^

build-test.cmd

@@ -224,7 +224,7 @@ set __MsbuildWrn=/flp1:WarningsOnly;LogFile="%__BuildWrn%"
 set __MsbuildErr=/flp2:ErrorsOnly;LogFile="%__BuildErr%"
 set __Logging=!__MsbuildLog! !__MsbuildWrn! !__MsbuildErr!
 
-call "%__ProjectDir%\msbuild.cmd" /nologo /verbosity:minimal /clp:Summary /nodeReuse:false^
+call "%__ProjectDir%\cmake_msbuild.cmd" /nologo /verbosity:minimal /clp:Summary /nodeReuse:false^
   /l:BinClashLogger,Tools/net46/Microsoft.DotNet.Build.Tasks.dll;LogFile=binclash.log^
   /p:RestoreDefaultOptimizationDataPackage=false /p:PortableBuild=true^
   /p:UsePartialNGENOptimization=false /maxcpucount^

build-test.sh

@@ -64,7 +64,6 @@ build_test_wrappers()
         __Logging="$__MsbuildLog $__MsbuildWrn $__MsbuildErr /consoleloggerparameters:$buildVerbosity"
 
         nextCommand="\"${__DotNetCli}\" msbuild \"${__ProjectDir}/tests/runtest.proj\" /p:RestoreAdditionalProjectSources=https://dotnet.myget.org/F/dotnet-core/ /p:BuildWrappers=true /p:TargetsWindows=false $__Logging /p:__BuildOS=$__BuildOS /p:__BuildType=$__BuildType /p:__BuildArch=$__BuildArch"
-        echo "$nextCommand"
         eval $nextCommand
 
         if [ $? -ne 0 ]; then
@@ -587,7 +586,7 @@ __SourceDir="$__ProjectDir/src"
 __PackagesDir="$__ProjectDir/packages"
 __RootBinDir="$__ProjectDir/bin"
 __BuildToolsDir="$__ProjectDir/Tools"
-__DotNetCli="${__BuildToolsDir}/dotnetcli/dotnet"
+__DotNetCli="$__ProjectDir/dotnet.sh"
 __UnprocessedBuildArgs=
 __CommonMSBuildArgs=
 __MSBCleanBuildArgs=
@@ -879,9 +878,6 @@ initTargetDistroRid
 
 # Override tool directory
 
-__CoreClrVersion=1.1.0
-__sharedFxDir=$__BuildToolsDir/dotnetcli/shared/Microsoft.NETCore.App/$__CoreClrVersion/
-
 if [[ (-z "$__GenerateLayoutOnly") && (-z "$__GenerateTestHostOnly") && (-z "$__BuildTestWrappersOnly") ]]; then
     build_Tests
 elif [ ! -z "$__BuildTestWrappersOnly" ]; then