__pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed. running DirectoryServices tests on Linux coreclr

[danmoseley]

Not much to go on --

Libraries Test Run release coreclr Linux x64 Debug
(Fedora.30.Amd64.Open)ubuntu.1604.amd64.open@mcr.microsoft.com/dotnet-buildtools/prereqs:fedora-30-helix-20200512010621-4f8cef7.

https://dev.azure.com/dnceng/public/_build/results?buildId=718537&view=logs&jobId=71afe768-e493-597c-228c-037834193aeb&j=71afe768-e493-597c-228c-037834193aeb&t=3e74a8b7-5e62-58f8-6a85-9e36ef39d1c1

    System.DirectoryServices.Protocols.Tests.DirectoryServicesProtocolsTests.TestAddUser [SKIP]
      Condition(s) not met: "IsLdapConfigurationExist"
dotnet: ../nptl/pthread_mutex_lock.c:79: __pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed.
./RunTests.sh: line 161:  2035 Aborted                 "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.DirectoryServices.Protocols.Tests.runtimeconfig.json --depsfile System.DirectoryServices.Protocols.Tests.deps.json xunit.console.dll System.DirectoryServices.Protocols.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
~/work/B45D097D/w/B56E0989/e
----- end Tue Jul 7 05:00:01 UTC 2020 ----- exit code 134 ----------------------------------------------------------
exit code 134 means SIGABRT Abort. Managed or native assert, or runtime check such as heap corruption, caused call to abort(). Core dumped.
Looking around for any Linux dump...
... found no dump in /home/helixbot/work/B45D097D/w/B56E0989/e

@davidfowl where are we at with collecting dump files? 😃

[danmoseley]

Note this was a PR job, I'm guessing the changes are not relevant, but here they are: https://github.com/dotnet/runtime/pull/38696/files

[danmoseley]

Uh, this assert is in glibc itself. I guess this is not actionable without a stack/dump/repro.

[danmoseley]

https://stackoverflow.com/questions/44635333/pthread-mutex-lock-assertion-fails

[jkotas]

This is most likely caused by a memory corruption caused by directory services code. Linux implementation of System.DirectoryServices.Protocols is very recent and it has a lot of unmanaged interop.

[danmoseley]

Ah -- good point. @joperezr .. maybe worth trying to stress these tests in this configuration?

[danmoseley]

Also if others hit this - please addref this issue, particularly if it's a different configuration. That could be a clue.

[stephentoub]

I've not seen that particular symptom, but I just got this, which could be releated if it's memory corruption:
https://dev.azure.com/dnceng/public/_build/results?buildId=723229&view=logs&jobId=73b7db51-6fbc-594d-25c9-15b2737b0bf6

/root/helix/work/workitem /root/helix/work/workitem
  Discovering: System.DirectoryServices.Protocols.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.DirectoryServices.Protocols.Tests (found 343 of 369 test cases)
  Starting:    System.DirectoryServices.Protocols.Tests (parallel test collections = on, max threads = 2)
./RunTests.sh: line 161:    22 Segmentation fault      (core dumped) "$RUNTIME_PATH/dotnet" exec --runtimeconfig System.DirectoryServices.Protocols.Tests.runtimeconfig.json --depsfile System.DirectoryServices.Protocols.Tests.deps.json xunit.console.dll System.DirectoryServices.Protocols.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing $RSP_FILE
/root/helix/work/workitem
----- end Thu Jul 9 21:07:22 UTC 2020 ----- exit code 139 ----------------------------------------------------------
exit code 139 means SIGSEGV Illegal memory access. Deref invalid pointer, overrunning buffer, stack overflow etc. Core dumped.