cmd/compile, runtime: investigate Windows stack overflows calling into system C libraries

[kjk]

This came up with 1.8.3 when working on GUI Windows programs using lxn\walk library.

Relevant issues:

• WebView crashes program with some websites lxn/walk#290
• WebView: fatal: morestack on g0 lxn/walk#261

The problem is that Go linker sets very small stack size (I think 128 kB) in PE executables. The standard on Windows is more like 1-2 MB.

This is fine for code that only lightly uses system C libraries but when writing code that talks to win32 UI APIs, it's very likely to hit the stack limit and silently crash.

I encountered it because I tried to use webview (mshtml.dll) control and it crashed on 64-bit when rendering my (not very complicated) website. Other people seen such crashes as well.

There are work-arounds: one can edit PE header after the exe is built using e.g. editbin.exe , but such tool is not necessarily available to the programmer (it's part of Visual Studio).

It would be much easier on Windows developers if there was a linker flag to set custom stack size so that it could be done directly with go build. A library like lxn\walk could then document the need to increase stack size for Go Windows programs and recommend a

I don't know if such option would be relevant/needed for other OSes/exe formats.

[bradfitz]

Stack sizes isn't a knob programmers should think about in Go. That's why Go stacks dynamically split or adjust in size as needed.

When making a call to code not managed by Go's dynamic stack size checks, Go should be switching to a conservatively-sized stack. Maybe there's a bug on how that works on Windows.

/cc @aclements @alexbrainman

[kjk]

For what it's worth, here's a callstack of the crash. It's on main thread:

.. and 0xe4 more stack frames likes this
e5 00000000`00088dd0 00007ffd`5eaacfcb MSHTML!CFormatInfo::FindFormattingParent+0x43a
e6 00000000`00088e20 00007ffd`5e99a90f MSHTML!CElement::ComputeFormatsVirtual+0x12b
e7 00000000`000897c0 00007ffd`5eab0f0a MSHTML!CElement::ComputeFormats+0x14f
e8 00000000`000898f0 00007ffd`5eaacfcb MSHTML!CFormatInfo::FindFormattingParent+0x43a
e9 00000000`00089940 00007ffd`5e99a90f MSHTML!CElement::ComputeFormatsVirtual+0x12b
ea 00000000`0008a2e0 00007ffd`5e999b61 MSHTML!CElement::ComputeFormats+0x14f
eb 00000000`0008a410 00007ffd`5e999765 MSHTML!CTreeNode::ComputeFormats+0x81
ec 00000000`0008a450 00007ffd`5e9ccde2 MSHTML!CTreeNode::ComputeFormatsHelper+0x35
ed 00000000`0008b200 00007ffd`5e9cde27 MSHTML!CTreeNode::GetCharFormatHelper+0xe
ee 00000000`0008b230 00007ffd`5ed6f105 MSHTML!CTreeNode::GetCharFormat+0x77
ef 00000000`0008b260 00007ffd`5ecd5226 MSHTML!CRecalcLinePtr::CalcAfterSpace+0x135
f0 00000000`0008b310 00007ffd`5ecc4c0e MSHTML!CRecalcLinePtr::MeasureLine+0x382
f1 00000000`0008b480 00007ffd`5ecc3375 MSHTML!CDisplay::RecalcLinesWithMeasurer+0x2ce
f2 00000000`0008b5e0 00007ffd`5ecda3d0 MSHTML!CDisplay::RecalcLines+0x65
f3 00000000`0008b820 00007ffd`5ecd7182 MSHTML!CDisplay::RecalcView+0x54
f4 00000000`0008b860 00007ffd`5ecd8196 MSHTML!CFlowLayout::CalcTextSize+0x302
f5 00000000`0008b9d0 00007ffd`5eddf0b5 MSHTML!CFlowLayout::CalcSizeCoreCSS1Strict+0xcc6
f6 00000000`0008bdc0 00007ffd`5ed4c2b4 MSHTML!CBodyLayout::CalcSizeCore+0x75
f7 00000000`0008be50 00007ffd`5ecc8af2 MSHTML!CFlowLayout::CalcSizeVirtual+0x84
f8 00000000`0008bee0 00007ffd`5ed48f67 MSHTML!CLayout::CalcSize+0x1da
f9 00000000`0008c080 00007ffd`5ecc8af2 MSHTML!CHtmlLayout::CalcSizeVirtual+0x9e7
fa 00000000`0008c3f0 00007ffd`5edf3efe MSHTML!CLayout::CalcSize+0x1da
fb 00000000`0008c590 00007ffd`5eae760f MSHTML!CLayout::CalcTopLayoutSize+0x5e
fc 00000000`0008c640 00007ffd`5ea3214d MSHTML!CView::EnsureSize+0xe7
fd 00000000`0008c680 00007ffd`5eae9e97 MSHTML!CView::EnsureView+0x43d
fe 00000000`0008c7a0 00007ffd`5ec60a6f MSHTML!CDoc::RunningToInPlace+0x1b7
ff 00000000`0008c8c0 00007ffd`5ec60754 MSHTML!CServer::TransitionTo+0xeb
100 00000000`0008c900 00007ffd`6d886ada MSHTML!CServer::Show+0x64
101 00000000`0008c930 00007ffd`6d884ce8 ieframe!CDocObjectHost::_ShowMsoView+0xba
102 00000000`0008c960 00007ffd`5eaebd4e ieframe!CDocObjectHost::ActivateMe+0x38
103 00000000`0008c990 00007ffd`5eaebcc7 MSHTML!CServer::ActivateView+0x72
104 00000000`0008c9c0 00007ffd`5ec6068f MSHTML!CServer::DoUIActivate+0x27
105 00000000`0008ca20 00007ffd`5ed682b5 MSHTML!CServer::DoVerb+0xaf
106 00000000`0008ca80 00007ffd`6d88681d MSHTML!CMarkup::Navigate+0x61
107 00000000`0008caf0 00007ffd`6d886dc0 ieframe!CDocObjectHost::_ActivateMsoView+0x91
108 00000000`0008cb90 00007ffd`6d888912 ieframe!CDocObjectHost::UIActivate+0x78
109 00000000`0008cbc0 00007ffd`6d8b0ad6 ieframe!CDocObjectView::UIActivate+0x22
10a 00000000`0008cbf0 00007ffd`6d8b21c1 ieframe!CBaseBrowser2::_UIActivateView+0xca
10b 00000000`0008cc20 00007ffd`6daa60c9 ieframe!CBaseBrowser2::v_ActivatePendingView+0x291
10c 00000000`0008ed70 00007ffd`6d9b82cb ieframe!CWebBrowserSB::v_ActivatePendingView+0x19
10d 00000000`0008eda0 00007ffd`6d8b2666 ieframe!`Microsoft::WRL::Module<1,Microsoft::WRL::Details::DefaultModule<1> >::Create'::`2'::`dynamic atexit destructor for 'moduleSingleton''+0x26e0b
10e 00000000`0008ee10 00007ffd`6daa1cf3 ieframe!CBaseBrowser2::Exec+0xd6
10f 00000000`0008ee70 00007ffd`6d9a6217 ieframe!CWebBrowserSB::Exec+0xd3
110 00000000`0008eef0 00007ffd`6d88273a ieframe!`Microsoft::WRL::Module<1,Microsoft::WRL::Details::DefaultModule<1> >::Create'::`2'::`dynamic atexit destructor for 'moduleSingleton''+0x14d570
111 00000000`0008ef60 00007ffd`6d8825a5 ieframe!CDocObjectHost::_OnReadyState+0x152
112 00000000`0008f1f0 00007ffd`6d8824c7 ieframe!CDocObjectHost::_OnChangedReadyState+0xd1
113 00000000`0008f2c0 00007ffd`5eba97d6 ieframe!CDocObjectHost::OnChanged+0x17
114 00000000`0008f2f0 00007ffd`5eba916d MSHTML!CBase::FirePropertyNotify+0x2f6
115 00000000`0008f390 00007ffd`5ead3cdc MSHTML!CMarkup::SetReadyState+0xfd
116 00000000`0008f3e0 00007ffd`5e960009 MSHTML!CMarkup::SetInteractiveInternal+0x36c
117 00000000`0008f720 00007ffd`5ead169d MSHTML!CMarkup::RequestReadystateInteractive+0xb5
118 00000000`0008f780 00007ffd`5ebbf9cc MSHTML!CMarkup::BlockScriptExecutionHelper+0x129
119 00000000`0008f7d0 00007ffd`5ead1abe MSHTML!CHtmPost::Exec+0x5dc
11a 00000000`0008f9d0 00007ffd`5ead199f MSHTML!CHtmPost::Run+0x32
11b 00000000`0008fa00 00007ffd`5ead185d MSHTML!PostManExecute+0x63
11c 00000000`0008fa40 00007ffd`5eb86780 MSHTML!PostManResume+0xa1
11d 00000000`0008fa80 00007ffd`5eb7042c MSHTML!CHtmPost::OnDwnChanCallback+0x40
11e 00000000`0008fad0 00007ffd`5ea97b71 MSHTML!CDwnChan::OnMethodCall+0x1c
11f 00000000`0008fb00 00007ffd`5ea90b39 MSHTML!GlobalWndOnMethodCall+0x251
120 00000000`0008fbb0 00007ffd`9a11bc50 MSHTML!GlobalWndProc+0xf9
121 00000000`0008fc40 00007ffd`9a11b5cf USER32!UserCallWinProcCheckWow+0x280
122 00000000`0008fda0 00000000`0045163e USER32!DispatchMessageWorker+0x19f
123 00000000`0008fe20 000000c0`42098d58 FindFilesGo+0x5163e
124 00000000`0008fe28 00000000`00790ca8 0x000000c0`42098d58
125 00000000`0008fe30 000000c0`42098cf0 FindFilesGo+0x390ca8
126 00000000`0008fe38 000000c0`42098cf0 0x000000c0`42098cf0
127 00000000`0008fe40 00000000`009d5be8 0x000000c0`42098cf0
128 00000000`0008fe48 00000000`00000000 0x9d5be8

Go code doing the windows message pump https://github.com/lxn/walk/blob/master/form.go#L336

[kjk]

For easy reproduction, I've created https://github.com/kjk/go20975

[aclements]

Stack sizes isn't a knob programmers should think about in Go.

The problem isn't in Go code.

The problem is that Go linker sets very small stack size (I think 128 kB) in PE executables. The standard on Windows is more like 1-2 MB.

Based on my reading of the code, the stack should be set to 2MB in cgo binaries. It's only 128KB in pure Go binaries. Could you double check what the PE header says?

[kjk]

0:000> !dh 400000 -f

File Type: EXECUTABLE IMAGE
FILE HEADER VALUES
    8664 machine (X64)
       C number of sections
       0 time date stamp
  4D4400 file pointer to symbol table
    2126 number of symbols
      F0 size of optional header
     223 characteristics
            Relocations stripped
            Executable
            App can handle >2gb addresses
            Debug information stripped

OPTIONAL HEADER VALUES
     20B magic #
    3.00 linker version
  32FE00 size of code
   1B600 size of initialized data
       0 size of uninitialized data
   51440 address of entry point
    1000 base of code
         ----- new -----
0000000000400000 image base
    1000 section alignment
     200 file alignment
       3 subsystem (Windows CUI)
    4.00 operating system version
    1.00 image version
    4.00 subsystem version
  56B000 size of image
     600 size of headers
       0 checksum
0000000000020000 size of stack reserve
0000000000001000 size of stack commit
0000000000100000 size of heap reserve
0000000000001000 size of heap commit

0x20000 == 131,072 bytes.

The thing is, this code doesn't use cgo. All calls to win libraries are via syscall.

[kjk]

The discussion in lxn/walk#261 claims that adding a dummy import _ "runtime/cgo" to the program would force the large stack but it doesn't seem to be the case in my testing. Maybe that changed sometime before 1.8.3.

[kjk]

Scratch that, it probably would have worked if I had gcc installed:

C:\Go\pkg\tool\windows_amd64\link.exe: running gcc failed: exec: "gcc": executable file not found in %PATH%

[alexbrainman]

I had quick look at how we set reserved stack size.

For Cgo program, it is 1M for 386 and 2M for amd64.
Search for oh64.SizeOfStackReserve and oh.SizeOfStackReserve in cmd/link/internal/ld/pe.go for first thread. All other threads start with

_beginthread(..., 0, ...)

in runtime/cgo/gcc_windows_386.c and runtime/cgo/gcc_windows_amd64.c, and _beginthread defaults to what first thread does.

For non-Cgo program, it is 128K.
Search for oh64.SizeOfStackReserve and oh.SizeOfStackReserve in cmd/link/internal/ld/pe.go for first thread. All other threads start with

stdcall6(_CreateThread, ..., 0x20000, ..., _STACK_SIZE_PARAM_IS_A_RESERVATION, ...)

So Cgo programs run with standard (by Windows standards) stacks. But non-Cgo programs run with small stacks. CL 2237 doubled the stack size (from 64K to 128K) already. @kjk can you, please, try make it 256K for non-Cgo - see if it fixes your problem. Maybe we can increase stack size again. I don't think it matters on amd64, and we have less and less of 386 computers.

Thank you

Alex

[kjk]

I tried 256kb, 512kb, 1mb, 2mb. My websites stops crashing at 512mb. Other websites need more. My vote would be for 2MB on win/64bit.

[alexbrainman]

My websites stops crashing at 512mb.

I was hoping it would work with 256K.

My vote would be for 2MB on win/64bit.

What about 386? Should we make stacks different on 386 and amd64?

Alex