[release/3.1] Update dependencies from dotnet/corefx

[dotnet-maestro]

This pull request updates the following dependencies

From https://github.com/dotnet/corefx

• Subscription: afd7026d-5288-4298-7366-08d72a4ce44a
• Build: 20210610.1
• Date Produced: 6/10/2021 10:15 PM
• Commit: 2395c5067189349691684542705cc3d9dc73bfe0
• Branch: refs/heads/release/3.1

• Updates:
  • Microsoft.Private.CoreFx.NETCoreApp: from 4.7.0-servicing.20460.1 to 4.7.0-servicing.21310.1

[wtgodbe]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[danmoseley]

Tests are totally broken on several configs, eg.,

File name: 'System.Runtime, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
 ---> System.IO.FileNotFoundException: Could not load file or assembly 'System.Runtime, Version=4.2.2.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. The system cannot find the file specified.

@wtgodbe @Anipik is this a mismatch in versioning between the repos somehow related to branding?

[wtgodbe]

Yeah, I think this will be unblocked once the update from core-setup is in

[wtgodbe]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[wtgodbe]

@danmoseley @jeffschwMSFT lots of tests failures here that look new, can someone take a look? e.g. System.Diagnostics.Tests.DebuggerDisplayAttributeTests.Target_SetNull_ThrowsArgumentNullException

[jeffschwMSFT]

@wtgodbe will do. I am seeing a lot of test build breaks. Do those fit within the expected? (I will ask that we follow-up to get those resolved).

[wtgodbe]

I don't remember seeing those build breaks before, though admittedly I'm not as familiar with this build infra as I used to be.

[jeffschwMSFT]

I have asked some folks to take a look. I would like to get this back to a clean state.

cc @hoyosjs @jkoritzinsky @trylek @agocke @tommcdon @JulieLeeMSFT

[jkoritzinsky]

Looks like the packages we're ingesting are not on the old dotnet-core feed. They're only on the 3.1 and 3.1-transport feeds. I've added those feeds to the test build so we can correctly set up Core_Root and the CoreFX test host.

[jeffschwMSFT]

Thanks @jkoritzinsky. The run has gotten further, but is still failing with missing dependencies.

Could not load file or assembly 'System.IO.Pipelines, Version=4.0.2.3, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.

Could not load file or assembly 'System.Drawing.Common, Version=4.0.2.2, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.

Are there more places that require a package update?

[jkoritzinsky]

IIRC those are both out-of-band packages, so it's probably a slightly different fix.

I'll continue investigating.

[jkoritzinsky]

So it looks like we aren't auto-updating those packages. Additionally, I don't know what feed they're on, so I don't know where to find the right version of those packages or what feeds I need to add.

@wtgodbe @Anipik any ideas for where to find the System.Drawing.Common and System.IO.Pipelines packages that include assemblies with matching assembly versions to the ones the CoreFX tests expect here?

[jeffschwMSFT]

Is an option to disable these tests? This is likely a repeated leg (given this is servicing).

[hoyosjs]

~ You'd need something like https://github.com/dotnet/coreclr/blob/release/3.1/tests/arm/corefx_linux_test_exclusions.txt, that disables the whole suite. ~ Looks like

coreclr/tests/CoreFX/CoreFX.issues.rsp

Line 24 in 5ddece2

-notrait category=RequiresElevation

would do what we need.

[hoyosjs]

This patch should be enough

diff --git a/tests/CoreFX/CoreFX.issues.rsp b/tests/CoreFX/CoreFX.issues.rsp
index 0d1a44452d8..82ca7f8abd7 100644
--- a/tests/CoreFX/CoreFX.issues.rsp
+++ b/tests/CoreFX/CoreFX.issues.rsp
@@ -89,3 +89,10 @@

 # Test failure: https://github.com/dotnet/runtime/issues/43166
 -nomethod System.IO.Tests.File_GetSetTimes.SetDateTimeMax
+
+# We have issues in restoring the OOB packages to run these tests. Disable for now.
+-nonamespace System.Data.Common.Tests
+-nonamespace System.IO.Pipelines.Tests
+-nonamespace System.Resources.Extensions.Tests
+-nonamespace System.Resources.ResourceManager.Tests
+-nonamespace System.Runtime.Serialization.Formatters.Tests

[wtgodbe]

@wtgodbe @Anipik any ideas for where to find the System.Drawing.Common and System.IO.Pipelines packages that include assemblies with matching assembly versions to the ones the CoreFX tests expect here?

I don't know of any reason they wouldn't be on dotnet3.1 or dotnet3.1-transport - @mmitche ?

[jeffschwMSFT]

Current failure (which looks transitory)

Failed to download 'https://dot.net/v1/dotnet-install.ps1'
GetDotNetInstallScript : Unable to connect to the remote server
At D:\workspace_work\1\s\eng\common\tools.ps1:222 char:20

• $installScript = GetDotNetInstallScript $dotnetRoot

•                ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  

  • CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
  • FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,GetDotNetInstallScript

Unable to download file in 5 attempts.
System.Management.Automation.RuntimeException: Unable to download file in 5 attempts.

[hoyosjs]

I already looped dnceng on that one: dotnet/runtime#55312. It's hitting all our branches.

[jeffschwMSFT]

thanks @jkoritzinsky and @hoyosjs - what is the next step? I see a few JIT failures and System.Drawing, etc

[jkoritzinsky]

The JIT tests are due to a network blip when downloading test assets.

The CoreFX tests are just more that we need to add to the exclusion (same issue with OOB System.Drawing.Common).

[hoyosjs]

@jkoritzinsky looks like the exclusions didn't work at all somehow. I'll download the payload.

[hoyosjs]

It feels like the issue is more like, the exclusions happen at the reflection level (enumeration happens, the type is in a namespace, sees it and gets excluded). Which means it needs to compile the test before, and in that process we get the loader exception. We have two possible routes here.

• If we see value to running the tests here, we can fix the test host.
• Otherwise, we can disable the corefx legs. They made a lot more sense in the dev days to prevent instability. Now, they are more brittle infra than they might actually catch regressions. As long as corefx tests run on top of the recent coreclr bits, this feel like a better way of not blocking future merges. I checked and that's the case for internal branches. The disadvantage is we only see these in the dependency update time, so only once the code changes get merged in CoreCLR and they flow upstack. CC: @Anipik @safern

[jeffschwMSFT]

we can disable the corefx legs.
As long as corefx tests run on top of the recent coreclr bits, this feel like a better way of not blocking future merges. I checked and that's the case for internal branches.
I agree @hoyosjs. Thanks for checking.

[hoyosjs]

The tracing r2r test is failing in several branches cc: @josalem

Also, the tracing test on x64 where the EventPipeProvider::m_pEventList is null and caused a segv on x64 linux.

00007FD8BFF45910 00007FD8C72FCD35 libcoreclr.so!SList<SListElem<EventPipeEvent*>, false, SListElem<EventPipeEvent*>*, &(SListElem<EventPipeEvent*>::m_Link)>::InsertTail(SListElem<EventPipeEvent*>*) + 85 at /__w/1/s/src/inc/slist.h:190                                                                                                                                                                                                
00007FD8BFF45940 00007FD8C72FC619 libcoreclr.so!EventPipeProvider::AddEvent(EventPipeEvent&) + 137 at /__w/1/s/src/vm/eventpipeprovider.cpp:195                                                                                                                                                                                                                                                                                           
00007FD8BFF459B0 00007FD8C72FC54E libcoreclr.so!EventPipeProvider::AddEvent(unsigned int, long, unsigned int, EventPipeEventLevel, bool, unsigned char*, unsigned int) + 174 at /__w/1/s/src/vm/eventpipeprovider.cpp:178                                                                                                                                                                                                                 
00007FD8BFF45A50 00007FD8C75CA4E8 libcoreclr.so!EventPipeInternal::DefineEvent(long, unsigned int, long, unsigned int, unsigned int, void*, unsigned int) + 552 at /__w/1/s/src/vm/eventpipeinternal.cpp:123                                                                                                                                                                                                                              
00007FD8BFF45BA0 00007FD84DAAFAB2                                                                                                                                                                                                                                                                                                                                                                                                         
00007FD8BFF45BB0                  [InlinedCallFrame: 00007fd8bff45bb0]                                                                                                                                                                                                                                                                                                                                                                    
00007FD8BFF45BB0                  [InlinedCallFrame: 00007fd8bff45bb0]                                                                                                                                                                                                                                                                                                                                                                    
00007FD8BFF45BA0 00007FD84DAAFA93 System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventPipeEventProvider.System.Diagnostics.Tracing.IEventProvider.DefineEventHandle(UInt32, System.String, Int64, UInt32, UInt32, Byte*, UInt32) + 83                                                                                                                                                                                              
00007FD8BFF45C60 00007FD84DACFBD8 System.Private.CoreLib.dll!System.Diagnostics.Tracing.NameInfo.GetOrCreateEventHandle(System.Diagnostics.Tracing.EventProvider, System.Diagnostics.Tracing.TraceLoggingEventHandleTable, System.Diagnostics.Tracing.EventDescriptor, System.Diagnostics.Tracing.TraceLoggingEventTypes) + 328                                                                                                           
00007FD8BFF45CE0 00007FD84DAC4066 System.Private.CoreLib.dll!System.Diagnostics.Tracing.EventSource.WriteImpl(System.String, System.Diagnostics.Tracing.EventSourceOptions ByRef, System.Object, System.Guid*, System.Guid*, System.Diagnostics.Tracing.TraceLoggingEventTypes) + 246                                                                                                                                                     
00007FD8BFF45E80 00007FD84DACD1C7 System.Private.CoreLib.dll!System.Diagnostics.Tracing.IncrementingPollingCounter.WritePayload(Single, Int32) + 695                                                                                                                                                                                                                                                                                      
00007FD8BFF45F50 00007FD84DAB6076 System.Private.CoreLib.dll!System.Diagnostics.Tracing.CounterGroup.OnTimer() + 342                                                                                                                                                                                                                                                                                                                      
00007FD8BFF46000 00007FD84DAB6541 System.Private.CoreLib.dll!System.Diagnostics.Tracing.CounterGroup+<>c__DisplayClass21_0.<PollForValues>b__0() + 17                                                                                                                                                                                                                                                                                     
00007FD8BFF46010 00007FD84DAB6479 System.Private.CoreLib.dll!System.Diagnostics.Tracing.CounterGroup.PollForValues() + 601                                                                                                                                                                                                                                                                                                                
00007FD8BFF460B0 00007FD84D949148 System.Private.CoreLib.dll!System.Threading.ThreadHelper.ThreadStart() + 152                                                                                                                                                                                                                                                                                                                            
00007FD8BFF460E0 00007FD8C74A4593 libcoreclr.so!CallDescrWorkerInternal + 124 at /__w/1/s/src/pal/inc/unixasmmacrosamd64.inc:870                                                                                                                                                                                                                                                                                                          
00007FD8BFF46100 00007FD8C7293570 libcoreclr.so!CallDescrWorkerWithHandler(CallDescrData*, int) + 400 at /__w/1/s/src/vm/callhelpers.cpp:70                                                                                                                                                                                                                                                                                               
00007FD8BFF46170 00007FD8C7294AB1 libcoreclr.so!MethodDescCallSite::CallTargetWorker(unsigned long const*, unsigned long*, int) + 2401 at /__w/1/s/src/vm/callhelpers.cpp:607                                                                                                                                                                                                                                                             
00007FD8BFF46480 00007FD8C70D8CB3 libcoreclr.so!MethodDescCallSite::Call(unsigned long const*) + 35 at /__w/1/s/src/vm/callhelpers.h:468                                                                                                                                                                                                                                                                                                  
00007FD8BFF464A0 00007FD8C72A79EF libcoreclr.so!ThreadNative::KickOffThread_Worker(void*) + 1055 at /__w/1/s/src/vm/comsynchronizable.cpp:246                                                                                                                                                                                                                                                                                             
00007FD8BFF466E8                  [GCFrame: 00007fd8bff466e8]                                                                                                                                                                                                                                                                                                                                                                             
00007FD8BFF46750 00007FD8C722DAD8 libcoreclr.so!ManagedThreadBase_DispatchInner(ManagedThreadCallState*) + 40 at /__w/1/s/src/vm/threads.cpp:7467                                                                                                                                                                                                                                                                                         
00007FD8BFF46770 00007FD8C722D3C5 libcoreclr.so!ManagedThreadBase_DispatchMiddle(ManagedThreadCallState*) + 149 at /__w/1/s/src/vm/threads.cpp:7510                                                                                                                                                                                                                                                                                       
00007FD8BFF46920 00007FD8C722D1FD libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::$_6::operator()(ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::TryArgs*) const::'lambda'(Param*)::operator()(Param*) const + 29 at /__w/1/s/src/vm/threads.cpp:7676                                                                                                                                                  
00007FD8BFF46940 00007FD8C722D02F libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::$_6::operator()(ManagedThreadBase_DispatchOuter(ManagedThreadCallState*)::TryArgs*) const + 111 at /__w/1/s/src/vm/threads.cpp:7676                                                                                                                                                                                             
00007FD8BFF469E0 00007FD8C7229FA5 libcoreclr.so!ManagedThreadBase_DispatchOuter(ManagedThreadCallState*) + 229 at /__w/1/s/src/vm/threads.cpp:7702                                                                                                                                                                                                                                                                                        
00007FD8BFF46A78                  [DebuggerU2MCatchHandlerFrame: 00007fd8bff46a78]                                                                                                                                                                                                                                                                                                                                                        
00007FD8BFF46AA0 00007FD8C722A0E7 libcoreclr.so!ManagedThreadBase_FullTransition(void (*)(void*), void*, UnhandledExceptionLocation) + 71 at /__w/1/s/src/vm/threads.cpp:7722                                                                                                                                                                                                                                                             
00007FD8BFF46B00 00007FD8C722A092 libcoreclr.so!ManagedThreadBase::KickOff(void (*)(void*), void*) + 34 at /__w/1/s/src/vm/threads.cpp:7758                                                                                                                                                                                                                                                                                               
00007FD8BFF46B20 00007FD8C72A8443 libcoreclr.so!ThreadNative::KickOffThread(void*) + 2083 at /__w/1/s/src/vm/comsynchronizable.cpp:363                                                                                                                                                                                                                                                                                                    
00007FD8BFF46D20 00007FD8C72210C9 libcoreclr.so!Thread::intermediateThreadProc(void*) + 169 at /__w/1/s/src/vm/threads.cpp:2109                                                                                                                                                                                                                                                                                                           
00007FD8BFF46D70 00007FD8C79E08A7 libcoreclr.so!CorUnix::CPalThread::ThreadEntry(void*) + 2535 at /__w/1/s/src/pal/src/thread/thread.cpp:1812                                                                                                                                                                                                                                                                                             
00007FD8BFF46F00 00007FD8C8DF26DB libpthread.so.0!start_thread + 219 at /build/glibc-S9d2JN/glibc-2.27/nptl/pthread_create.c:463                                                                                                                                                                                                                                                                                                          
00007FD8BFF46FC0 00007FD8C81DC71F libc.so.6!__GI___clone + 63 at /build/glibc-S9d2JN/glibc-2.27/sysdeps/unix/sysv/linux/x86_64/clone.S:97

[josalem]

@hoyosjs that failure looks a lot like dotnet/runtime#55240 which also happened on R2R arm Linux. I need to look at the dump to be sure, but it's possible this is a regression caused by #28180 since it modified this code path. It is not obvious how or why that would be the case though.

The other test failures could also be happening due to this change since they are counter related. I'm not sure why they weren't failing before when we were adding the change though. These tests were passing before and during the PR. Let me investigate it.

[josalem]

Looking through the code and the dump, it looks like there is an issue on shutdown where a write is happening after an eventsource has been disabled and disposed. The special session for counters has been disabled and RuntimeEventSource has been disposed. The CountersdGroup and the RuntimeEventSource are in the finalizer queue. It looks like there isn't any other managed code running.

Here's my current hunch: there is a pre-existing (pre-#28180) issue with EventSource.Dispose().

coreclr/src/System.Private.CoreLib/shared/System/Diagnostics/Tracing/EventSource.cs

Lines 1350 to 1390 in 09f5b75

	protected virtual void Dispose(bool disposing)
	{
	if (disposing)
	{
	#if FEATURE_MANAGED_ETW
	// Send the manifest one more time to ensure circular buffers have a chance to get to this information
	// even in scenarios with a high volume of ETW events.
	if (m_eventSourceEnabled)
	{
	try
	{
	SendManifest(m_rawManifest);
	}
	catch (Exception)
	{ } // If it fails, simply give up.
	m_eventSourceEnabled = false;
	}
	if (m_etwProvider != null)
	{
	m_etwProvider.Dispose();
	m_etwProvider = null!; // TODO-NULLABLE: Avoid nulling out in Dispose
	}
	#endif
	#if FEATURE_PERFTRACING
	if (m_eventPipeProvider != null)
	{
	m_eventPipeProvider.Dispose();
	m_eventPipeProvider = null!; // TODO-NULLABLE: Avoid nulling out in Dispose
	}
	#endif
	}
	m_eventSourceEnabled = false;
	m_eventSourceDisposed = true;
	}
	/// <summary>
	/// Finalizer for EventSource
	/// </summary>
	~EventSource()
	{
	this.Dispose(false);
	}

This looks to be a race on shutdown path that the old code with the infinite loop issue was hiding. There is an exceedingly small window of time during the EventSource.Dispose() method where the source hasn't been marked disabled, but the internals have been disposed. In this case, the counters continue to try writing payloads and if the race happens, they attempt to write to an EventPipeEventProvider after free. The infinite loop bug would actually protect against this race from happening because it would stop the counters from writing in this case. This leads to a use after free caused by m_EventSourceEnabled not being set to false at the top of dispose. It affects 3.1, 5.0, and 6.0. The change for polling interval on counters made it so the runtime counters hit this window with higher frequency.

The dispose method on EventSource only gets called fully when the ~EventSource destructor is called.

I'll continue investigating this to be sure.

CC @noahfalk

[jeffschwMSFT]

Thanks @josalem. Can you create an issue to track the rare race condition?

[josalem]

Yes, I'll create it when I get back to my desk.

[safern]

Otherwise, we can disable the corefx legs. They made a lot more sense in the dev days to prevent instability. Now, they are more brittle infra than they might actually catch regressions. As long as corefx tests run on top of the recent coreclr bits, this feel like a better way of not blocking future merges. I checked and that's the case for internal branches. The disadvantage is we only see these in the dependency update time, so only once the code changes get merged in CoreCLR and they flow upstack. CC: @Anipik @safern

Completely agreed for servicing purposes.