Assert failure: !"Heap contamination detected! HeapFree was called on a heap other than the one that memory was allocated from.\n"

[BruceForstall]

System.Net.Requests.Tests and System.Security.Cryptography.Pkcs.Tests

Many jobs, x64/arm64/arm32

https://dev.azure.com/dnceng/public/_build/results?buildId=1232250&view=ms.vss-test-web.build-test-results-tab

+ grep COMPlus
COMPlus_TieredCompilation=0
COMPlus_DbgMiniDumpName=/home/helixbot/dotnetbuild/dumps/coredump.%d.dmp
COMPlus_DbgEnableMiniDump=1
+ ./RunTests.sh --runtime-path /datadisks/disk1/work/C2180A37/p
----- start Mon Jul 12 08:04:14 UTC 2021 =============== To repro directly: =====================================================
pushd .
/datadisks/disk1/work/C2180A37/p/dotnet exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
/datadisks/disk1/work/C2180A37/w/B3D40A47/e /datadisks/disk1/work/C2180A37/w/B3D40A47/e
  Discovering: System.Net.Requests.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Net.Requests.Tests (found 334 of 353 test cases)
  Starting:    System.Net.Requests.Tests (parallel test collections = on, max threads = 2)
    System.Net.Tests.FtpWebRequestTest.Ftp_AppendFile [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_RenameFile [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_LargeFile [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_CreateAndDelete [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_RenameFileSubDir_Success [SKIP]
      Condition(s) not met: "LocalServerAvailable"
    System.Net.Tests.FtpWebRequestTest.Ftp_MakeAndRemoveDir_Success [SKIP]
      Condition(s) not met: "LocalServerAvailable"

Assert failure(PID 24366 [0x00005f2e], Thread: 24381 [0x5f3d]): !"Heap contamination detected! HeapFree was called on a heap other than the one that memory was allocated from.\n" "Possible cause: you used new (executable) to allocate the memory, but didn't use DeleteExecutable() to free it."
    File: /__w/1/s/src/coreclr/utilcode/clrhost_nodependencies.cpp Line: 296
    Image: /datadisks/disk1/work/C2180A37/p/dotnet

Gathering state for process 24366 dotnet
Crashing thread 00005f3d signal 00000006
Writing minidump with heap to file /home/helixbot/dotnetbuild/dumps/coredump.24366.dmp
Written 271179776 bytes (66206 pages) to core file
Dump successfully written
./RunTests.sh: line 162: 24366 Aborted                 (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.Net.Requests.Tests.runtimeconfig.json --depsfile System.Net.Requests.Tests.deps.json xunit.console.dll System.Net.Requests.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/datadisks/disk1/work/C2180A37/w/B3D40A47/e
----- end Mon Jul 12 08:04:38 UTC 2021 ----- exit code 134 ----------------------------------------------------------

[BruceForstall]

@mangod9

[mangod9]

If this is a consistent failure for specific test category, might make sense for the owners to investigate if there is a smaller repro and if any allocations/pinvokes are causing a corruption.

[mangod9]

@karelz can some from system.net please take an initial look at the failures to determine if they might be corruption due to pinvokes here?

[stephentoub]

Again here:
https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-55499-merge-641ae0c0c3d241e0ac/System.Net.Requests.Tests/console.f66f1627.log?sv=2019-07-07&se=2021-08-02T02%3A22%3A55Z&sr=c&sp=rl&sig=oSYpPpupPfXa2ifp7JLYN4pM1Uk0xdYo8h%2B4xSxau90%3D

[mangod9]

Adding @janvorli as well in case this is related to the w^x work, assuming this started happening in the past day or so.

[janvorli]

I've found the issue, I'll create a PR with a fix in a minute.

[karelz]

Thanks @janvorli! Out of curiosity, what caused the problems? Was it w^x work or some churn in Networking/crypto?

[janvorli]

W^X work, I've forgotten to replace one call to c++ delete by an appropriate executable heap method.

[janvorli]

And it was in a place that's executed when we allocate a data structure from multiple threads and only one of them wins so the others free what they've allocated. That's why it was intermittent and I haven't caught it locally.