Fix underlying problem and enable UseSharedCompilation if possible (Roslyn shared server)

[dagood]

When testing a 3.0 tarball, I hit a problem with the shared Roslyn compiler server:

/home/dagood/sb/tarball_511741/packages/restored/microsoft.net.compilers.toolset/3.3.1/tasks/netcoreapp2.1/Microsoft.CSharp.Core.targets(59,5): Roslyn compiler server reports different hash version than build task. [/home/dagood/sb/tarball_511741/src/coreclr.259ce7d4619478cfefe7b0c0f6fa765f765f7e37/src/System.Private.CoreLib/System.Private.CoreLib.csproj]

A few of my notes from a convo:

Looking at microsoft.net.compilers.toolset... it looks like it contains the compiler. Before Roslyn, the DLL has commit 66a912c9463eebe832cf742d2fe8bb2e1a4600ec. After Roslyn, we delete the package from the cache and the new restored package with same version is 39d545dc14bc4d5a546e5e2f58d1ebaca72fcd61.

I wonder if my build runs too fast, and I'm getting unlucky that the roslyn server used before Roslyn is still up by the time I try to build CoreCLR, and at that point it detects the mismatch.

I wonder if the server would be smart enough to die if we used per-repo package caches so the server isn't confused by the file being changed from under it--maybe it stops gracefully if its compiler DLL is completely gone. Something to look into if turning off shared compilation is a significant perf hit or something.

The MSBuild property UseSharedCompilation controls whether this feature is active.

[dagood]

Triage notes:
We have similar behavior for MSBuild persistent nodes, and historically have always turned off behavior like this. Closing, no need to go after it right now.

[omajid]

FWIW, I am seeing the same problem in 3.0, except in corefx, in some of my build systems.

[dagood]

Opened #1527 to track the opposite of this issue: removing UseSharedCompilation everywhere to proactively stop running into this.