Seg fault in System.Drawing.Common tests on Unix

[stephentoub]

We've been plagued by seg faults on Linux in the System.Drawing.Common tests, e.g. this one on SUSE:
https://mc.dot.net/#/user/stephentoub/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fcli~2F/91ec984d64908b3ab312bef6f6fa599f5ea1cee7/workItem/System.Drawing.Common.Tests/wilogs

2017-10-09 19:45:10,371: INFO: proc(54): run_and_log_output: Output: Discovering: System.Drawing.Common.Tests
2017-10-09 19:45:12,085: INFO: proc(54): run_and_log_output: Output: Discovered:  System.Drawing.Common.Tests
2017-10-09 19:45:12,654: INFO: proc(54): run_and_log_output: Output: Starting:    System.Drawing.Common.Tests
2017-10-09 19:45:13,601: INFO: proc(54): run_and_log_output: Output:    System.Drawing.Printing.Tests.PrinterSettingsTests.MaximumCopies_ReturnsExpected [SKIP]
2017-10-09 19:45:13,601: INFO: proc(54): run_and_log_output: Output:       Condition(s) not met: \"IsAnyInstalledPrinters\"
2017-10-09 19:45:13,604: INFO: proc(54): run_and_log_output: Output:    System.Drawing.Printing.Tests.PrinterSettingsTests.LandscapeAngle_ReturnsExpected [SKIP]
2017-10-09 19:45:13,604: INFO: proc(54): run_and_log_output: Output:       Condition(s) not met: \"IsAnyInstalledPrinters\"
2017-10-09 19:45:13,606: INFO: proc(54): run_and_log_output: Output:    System.Drawing.Printing.Tests.PrinterSettingsTests.Collate_Default_ReturnsExpected [SKIP]
2017-10-09 19:45:13,606: INFO: proc(54): run_and_log_output: Output:       Condition(s) not met: \"IsAnyInstalledPrinters\"
2017-10-09 19:45:13,620: INFO: proc(54): run_and_log_output: Output:    System.Drawing.Printing.Tests.PrinterSettingsTests.IsPlotter_ReturnsExpected [SKIP]
2017-10-09 19:45:13,620: INFO: proc(54): run_and_log_output: Output:       Condition(s) not met: \"IsAnyInstalledPrinters\"
2017-10-09 19:45:13,629: INFO: proc(54): run_and_log_output: Output:    System.Drawing.Printing.Tests.PrinterSettingsTests.Static_InstalledPrinters_ReturnsExpected [SKIP]
2017-10-09 19:45:13,629: INFO: proc(54): run_and_log_output: Output:       Condition(s) not met: \"IsAnyInstalledPrinters\"
2017-10-09 19:45:14,110: INFO: proc(54): run_and_log_output: Output:    MonoTests.System.Drawing.Imaging.PngCodecTest.Bitmap2bitData [SKIP]
2017-10-09 19:45:14,110: INFO: proc(54): run_and_log_output: Output:       Condition(s) not met: \"GetRecentGdiPlusIsAvailable2\"
2017-10-09 19:45:14,110: INFO: proc(54): run_and_log_output: Output:    MonoTests.System.Drawing.Imaging.PngCodecTest.Bitmap2bitFeatures [SKIP]
2017-10-09 19:45:14,110: INFO: proc(54): run_and_log_output: Output:       Condition(s) not met: \"GetRecentGdiPlusIsAvailable2\"
2017-10-09 19:45:14,111: INFO: proc(54): run_and_log_output: Output:    MonoTests.System.Drawing.Imaging.PngCodecTest.Bitmap2bitPixels [SKIP]
2017-10-09 19:45:14,111: INFO: proc(54): run_and_log_output: Output:       Condition(s) not met: \"GetRecentGdiPlusIsAvailable2\"
2017-10-09 19:45:15,499: INFO: proc(54): run_and_log_output: Output: /home/helixbot/dotnetbuild/work/c092864f-24dd-40c6-a4ad-db55569616e0/Work/af3a10a9-a3b8-4472-893f-3ef12b40a6c4/Unzip/RunTests.sh: line 87:  4576 Segmentation fault      (core dumped) $RUNTIME_PATH/dotnet xunit.console.netcore.exe System.Drawing.Common.Tests.dll -xml testResults.xml -notrait Benchmark=true -notrait category=nonnetcoreapptests -notrait category=nonlinuxtests -notrait category=OuterLoop -notrait category=failing
2017-10-09 19:45:15,540: INFO: proc(54): run_and_log_output: Output: Trying to find crash dumps for project: System.Drawing.Common.Tests
2017-10-09 19:45:15,540: INFO: proc(54): run_and_log_output: Output: No new dump file was found in /home/helixbot/dotnetbuild/work/c092864f-24dd-40c6-a4ad-db55569616e0/Work/af3a10a9-a3b8-4472-893f-3ef12b40a6c4/Unzip
2017-10-09 19:45:15,542: INFO: proc(54): run_and_log_output: Output: ~/dotnetbuild/work/c092864f-24dd-40c6-a4ad-db55569616e0/Work/af3a10a9-a3b8-4472-893f-3ef12b40a6c4/Unzip
2017-10-09 19:45:15,543: INFO: proc(54): run_and_log_output: Output: Finished running tests. End time=19:45:15. Return value was 139
2017-10-09 19:45:15,544: INFO: proc(58): run_and_log_output: Exit Code: 139
2017-10-09 19:45:15,545: ERROR: scriptrunner(87): _main: Error: No exception thrown, but XUnit results not created
2017-10-09 19:45:15,545: ERROR: helix_test_execution(83): report_error: Error running xunit None

This has been happening frequently for months, but I can't find an existing issue for it.

[Drawaes]

Tonight was the night I planned to launch an issue for this. It seems a constant source of outerloop failures. And causes me untold annoyance and investigation ... :) fix or disable I say ;)

[danmoseley]

Unfortunately Dumpling isn't finding dumps on SUSE. @DrewScoggins have you had a chance ot take a look?
We need a dumpling upload from another distro.

[benaadams]

Related? https://github.com/dotnet/corefx/issues/24538

[danmoseley]

@benaadams I do'nt think so, that's freebsd specific.

[danmoseley]

@qmfrederik do you have any context on these segfaults? I'm not sure whether they're only on SUSE, but perhaps it has an outdated copy of libgdiplus (it doesn't return false from GetRecentGdiPlusIsAvailable())

/cc @safern

[danmoseley]

Debian8.7 now.

https://mc.dot.net/#/user/adiaaida/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fcli~2F/52c33af0e09a614ab89ba8b519fc21de0e53bdcf/workItem/System.Drawing.Common.Tests/wilogs

This should have everything needed to debug:
https://dumpling.azurewebsites.net/api/dumplings/archived/eefcb1cc36977ccf86f457ee28a33a7b4cc24e13

[qmfrederik]

@danmosemsft I don't have any immediate context for these segfaults.

Debian Jessie ships with version 4.2 of libgdiplus which dates from December 2015. There have been a few fixes to libdiplus since.

We'll probably need to do some work to identify all the areas where the corefx CI process identifies areas of libgdiplus which are not stable, and then decide how to deal with that; a couple of options include:

• Do nothing and let the end users discover the issues one by one
• Fix the issues in libgdiplus, and require that you use System.Drawing.Common with a recent version of libgdiplus
• Proactively throw PNSE's in System.Drawing.Common on Unix on areas where we know libgdiplus to be unstable.

Meanwhile, I've downloaded the dumpling but not entirely sure what I'm looking at. Is there a way to extract the managed and native stack traces from those dumpling files?

[Drawaes]

Is it possible to disable these tests on these platforms and tag them with this issue in the meantime then?

It sounds like a fix isn't a quick thing and they cause a lot of outerloop failures.

[qmfrederik]

@Drawaes Part of the problem is that for tests that cause the test process to crash (e.g. segfaults), there is no clear indication in the test logs which unit test caused the segfault - so it's not easy to say which test should be disabled (other than disabling all System.Drawing.Common tests all together).

[Drawaes]

Process of elimination then... you are on a binary search.

Remove 1 test on 1 distro that fails a lot. See if it starts passing, Keep going until that distro either has no tests left or passes. Once it passes go back and try to add the other tests and see if it all works. :)

At least you can narrow it hopefully, and secondly everyone else doesn't have to go into every outerloop because they always fail due to System.Drawing.

[stephentoub]

This one's on Fedora:
https://mc.dot.net/#/user/dotnet-maestro-bot/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fcli~2F/231901a598b4bb3c7591db6b3496a656c92e3644

[stephentoub]

And on CentOS:
https://mc.dot.net/#/user/dotnet-maestro-bot/pr~2Fjenkins~2Fdotnet~2Fcorefx~2Fmaster~2F/test~2Ffunctional~2Fcli~2F/231901a598b4bb3c7591db6b3496a656c92e3644

[safern]

Finally, we’ve got a dump, yay! @qmfrederik I sword of know how to open it, so I will try it in my Ubuntu box and update the thread. I think there is a doc if you want to try opening it in your end also. Let me find it an I’ll post it here.

[safern]

From the dumpling that @danmosemsft pointed out to I got this stack trace:

00007FDBFAAAF990 00007FDC2266A868 DomainBoundILStubClass.IL_STUB_PInvoke(IntPtr, IntPtr, Int32, Int32, Int32, Int32)
00007FDBFAAAFA30 00007FDC22807339 System.Drawing.SafeNativeMethods+Gdip.GdipDrawRectangleI(IntPtr, IntPtr, Int32, Int32, Int32, Int32)
00007FDBFAAAFA80 00007FDC2280726A System.Drawing.Graphics.DrawRectangle(System.Drawing.Pen, Int32, Int32, Int32, Int32)
00007FDBFAAAFAA0 00007FDC228071EB System.Drawing.Graphics.DrawRectangle(System.Drawing.Pen, System.Drawing.Rectangle)
00007FDBFAAAFAD0 00007FDC22802807 System.Drawing.Tests.RegionTests.Complement_GraphicsPathWithMultipleRectangles_Success()

Which points us to the test that crashed and the path that it followed to cause the crash. Will go ahead and take a look at other dumps (if there are) in the other build failures to see if there is other crashing test, and then I'll disable the crashing tests.

In the meantime, @qmfrederik would you mind taking a look at why this is crashing?

Here are the docs on how to load the dump: https://github.com/dotnet/coreclr/blob/master/Documentation/building/debugging-instructions.md#debugging-core-dumps-with-lldb

Let me know if I can help you.

[Drawaes]

Thanks that would be awesome