Flaky or bugged async-hooks/test-zlib.zlib-binding.deflate

[apapirovski]

For a good while now, this test has been flaky or perhaps broken on a number of the Linux platforms. Example failure:

00:25:23 not ok 57 async-hooks/test-zlib.zlib-binding.deflate
00:25:23   ---
00:25:23   duration_ms: 0.165
00:25:23   severity: crashed
00:25:23   exitcode: -11
00:25:23   stack: |-

but also the more thorough:

*** Error in `out/Release/node': malloc(): smallbin double linked list corrupted: 0x0000000011d104b0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x8c9d2)[0x3ff87a8c9d2]
/lib64/libc.so.6(+0x903b0)[0x3ff87a903b0]
/lib64/libc.so.6(__libc_malloc+0x70)[0x3ff87a92850]
/data/gcc-4.9/lib64/libstdc++.so.6(_Znwm+0x34)[0x3ff87e7d994]
out/Release/node(_ZNSt6vectorIN2v88internal10ParserBaseINS1_6ParserEE24DeclarationParsingResult11DeclarationESaIS6_EE19_M_emplace_back_auxIIRKS6_EEEvDpOT_+0x5e)[0x1d7e59e]
out/Release/node(_ZN2v88internal10ParserBaseINS0_6ParserEE25ParseVariableDeclarationsENS3_26VariableDeclarationContextEPNS3_24DeclarationParsingResultEPNS0_8ZoneListIPKNS0_12AstRawStringEEEPb+0xb60)[0x1d88788]
out/Release/node(_ZN2v88internal10ParserBaseINS0_6ParserEE17ParseForStatementEPNS0_8ZoneListIPKNS0_12AstRawStringEEEPb+0x42c)[0x1d9bac4]
out/Release/node(_ZN2v88internal10ParserBaseINS0_6ParserEE14ParseStatementEPNS0_8ZoneListIPKNS0_12AstRawStringEEENS0_30AllowLabelledFunctionStatementEPb+0x2b8)[0x1d83c48]
out/Release/node(_ZN2v88internal10ParserBaseINS0_6ParserEE18ParseStatementListEPNS0_8ZoneListIPNS0_9StatementEEENS0_5Token5ValueEbPb+0x9e)[0x1d83ffe]
out/Release/node(_ZN2v88internal10ParserBaseINS0_6ParserEE17ParseFunctionBodyEPNS0_8ZoneListIPNS0_9StatementEEEPKNS0_12AstRawStringEiRKNS0_22ParserFormalParametersENS0_12FunctionKindENS0_15FunctionLiteral12FunctionTypeEPb+0x10e)[0x1d85006]
out/Release/node(_ZN2v88internal6Parser13ParseFunctionEPKNS0_12AstRawStringEiNS0_12FunctionKindENS0_15FunctionLiteral12FunctionTypeEPNS0_16DeclarationScopeEPiSA_PbSA_SA_PNS0_8ZoneListIS4_EESB_+0x2c0)[0x1d89808]
out/Release/node(_ZN2v88internal6Parser20ParseFunctionLiteralEPKNS0_12AstRawStringENS0_7Scanner8LocationENS0_20FunctionNameValidityENS0_12FunctionKindEiNS0_15FunctionLiteral12FunctionTypeENS0_12LanguageModeEPNS0_8ZoneListIS4_EEPb+0x464)[0x1d8a224]
out/Release/node(_ZN2v88internal6Parser15DoParseFunctionEPNS0_9ParseInfoEPKNS0_12AstRawStringE+0x16c)[0x1d8ef64]
out/Release/node(_ZN2v88internal6Parser13ParseFunctionEPNS0_7IsolateEPNS0_9ParseInfoENS0_6HandleINS0_18SharedFunctionInfoEEE+0x252)[0x1d8f992]
out/Release/node(_ZN2v88internal7parsing13ParseFunctionEPNS0_9ParseInfoENS0_6HandleINS0_18SharedFunctionInfoEEEPNS0_7IsolateE+0x14a)[0x1d9cf7a]
out/Release/node(_ZN2v88internal8Compiler7CompileENS0_6HandleINS0_18SharedFunctionInfoEEENS1_18ClearExceptionFlagE+0x1f4)[0x1aac4ac]
out/Release/node(_ZN2v88internal8Compiler7CompileENS0_6HandleINS0_10JSFunctionEEENS1_18ClearExceptionFlagE+0x230)[0x1ab1d30]
out/Release/node(_ZN2v88internal19Runtime_CompileLazyEiPPNS0_6ObjectEPNS0_7IsolateE+0x9c)[0x1e6c634]
[0x9507e84474]
======= Memory map: ========
01000000-03086000 r-xp 00000000 5e:05 1057338                            /data/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel72-s390x/out/Release/node
03087000-03089000 r--p 02086000 5e:05 1057338                            /data/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel72-s390x/out/Release/node
03089000-030a0000 rw-p 02088000 5e:05 1057338                            /data/iojs/build/workspace/node-test-commit-linuxone/nodes/rhel72-s390x/out/Release/node
030a0000-030b6000 rw-p 00000000 00:00 0
11c80000-11da5000 rw-p 00000000 00:00 0                                  [heap]
17dfa80000-17dfb00000 rw-p 00000000 00:00 0
2516100000-2516180000 rw-p 00000000 00:00 0
4a2cd80000-4a2ce00000 rw-p 00000000 00:00 0
533a380000-533a400000 rw-p 00000000 00:00 0
6171080000-6171100000 rw-p 00000000 00:00 0
648e000000-648e080000 rw-p 00000000 00:00 0
6a4a280000-6a4a300000 rw-p 00000000 00:00 0
6af8280000-6af8300000 rw-p 00000000 00:00 0
6b60d80000-6b60e00000 rw-p 00000000 00:00 0
8bf1600000-8bf1680000 rw-p 00000000 00:00 0
8fb8380000-8fb8400000 rw-p 00000000 00:00 0
9507e61000-9507e80000 ---p 00000000 00:00 0
9507e80000-9507e83000 rw-p 00000000 00:00 0
9507e83000-9507e84000 ---p 00000000 00:00 0
9507e84000-9507eff000 rwxp 00000000 00:00 0
9507eff000-9507f00000 ---p 00000000 00:00 0
9507f00000-9507f03000 rw-p 00000000 00:00 0
9507f03000-9507f04000 ---p 00000000 00:00 0
9507f04000-9507f7f000 rwxp 00000000 00:00 0
9507f7f000-9507f80000 ---p 00000000 00:00 0
9507f80000-9507f83000 rw-p 00000000 00:00 0
9507f83000-9507f84000 ---p 00000000 00:00 0
9507f84000-9507fff000 rwxp 00000000 00:00 0
9507fff000-9527e61000 ---p 00000000 00:00 0
959d180000-959d185000 rw-p 00000000 00:00 0
a5c0500000-a5c054a000 rw-p 00000000 00:00 0
bdcfced000-bdcfcf0000 ---p 00000000 00:00 0
bdcfcf0000-bdcfcf8000 rw-p 00000000 00:00 0
bdcfcf8000-bdcfcf9000 ---p 00000000 00:00 0
c0a8880000-c0a8900000 rw-p 00000000 00:00 0
caccf80000-cacd000000 rw-p 00000000 00:00 0
ce0d480000-ce0d500000 rw-p 00000000 00:00 0
da4c200000-da4c280000 rw-p 00000000 00:00 0
f33e100000-f33e180000 rw-p 00000000 00:00 0
3ff6c000000-3ff6c021000 rw-p 00000000 00:00 0
3ff6c021000-3ff70000000 ---p 00000000 00:00 0
3ff70000000-3ff70021000 rw-p 00000000 00:00 0
3ff70021000-3ff74000000 ---p 00000000 00:00 0
3ff77800000-3ff77801000 ---p 00000000 00:00 0
3ff77801000-3ff78000000 rw-p 00000000 00:00 0
3ff78000000-3ff78021000 rw-p 00000000 00:00 0
3ff78021000-3ff7c000000 ---p 00000000 00:00 0
3ff7c000000-3ff7c021000 rw-p 00000000 00:00 0
3ff7c021000-3ff80000000 ---p 00000000 00:00 0
3ff80000000-3ff80021000 rw-p 00000000 00:00 0
3ff80021000-3ff84000000 ---p 00000000 00:00 0
3ff84200000-3ff84201000 ---p 00000000 00:00 0
3ff84201000-3ff84a00000 rw-p 00000000 00:00 0
3ff84a00000-3ff84a01000 ---p 00000000 00:00 0
3ff84a01000-3ff85200000 rw-p 00000000 00:00 0
3ff85200000-3ff85201000 ---p 00000000 00:00 0
3ff85201000-3ff85a00000 rw-p 00000000 00:00 0
3ff85a00000-3ff85a01000 ---p 00000000 00:00 0
3ff85a01000-3ff86200000 rw-p 00000000 00:00 0
3ff86200000-3ff86201000 ---p 00000000 00:00 0
3ff86201000-3ff86a00000 rw-p 00000000 00:00 0
3ff86a00000-3ff86a01000 ---p 00000000 00:00 0
3ff86a01000-3ff87200000 rw-p 00000000 00:00 0
3ff87200000-3ff87201000 ---p 00000000 00:00 0
3ff87201000-3ff87a00000 rw-p 00000000 00:00 0
3ff87a00000-3ff87ba6000 r-xp 00000000 5e:02 5039                         /usr/lib64/libc-2.17.so
3ff87ba6000-3ff87baa000 r--p 001a5000 5e:02 5039                         /usr/lib64/libc-2.17.so
3ff87baa000-3ff87bad000 rw-p 001a9000 5e:02 5039                         /usr/lib64/libc-2.17.so
3ff87bad000-3ff87bb0000 rw-p 00000000 00:00 0
3ff87c00000-3ff87c18000 r-xp 00000000 5e:02 5071                         /usr/lib64/libpthread-2.17.so
3ff87c18000-3ff87c19000 r--p 00017000 5e:02 5071                         /usr/lib64/libpthread-2.17.so
3ff87c19000-3ff87c1a000 rw-p 00018000 5e:02 5071                         /usr/lib64/libpthread-2.17.so
3ff87c1a000-3ff87c1e000 rw-p 00000000 00:00 0
3ff87c80000-3ff87c92000 r-xp 00000000 5e:05 6423382                      /data/gcc-4.9/lib64/libgcc_s.so.1
3ff87c92000-3ff87c93000 r--p 00011000 5e:05 6423382                      /data/gcc-4.9/lib64/libgcc_s.so.1
3ff87c93000-3ff87c94000 rw-p 00012000 5e:05 6423382                      /data/gcc-4.9/lib64/libgcc_s.so.1
3ff87d00000-3ff87da8000 r-xp 00000000 5e:02 29820                        /usr/lib64/libm-2.17.so
3ff87da8000-3ff87da9000 r--p 000a7000 5e:02 29820                        /usr/lib64/libm-2.17.so
3ff87da9000-3ff87daa000 rw-p 000a8000 5e:02 29820                        /usr/lib64/libm-2.17.so
3ff87e00000-3ff87f30000 r-xp 00000000 5e:05 6424618                      /data/gcc-4.9/lib64/libstdc++.so.6.0.20
3ff87f30000-3ff87f31000 ---p 00130000 5e:05 6424618                      /data/gcc-4.9/lib64/libstdc++.so.6.0.20
3ff87f31000-3ff87f37000 r--p 00130000 5e:05 6424618                      /data/gcc-4.9/lib64/libstdc++.so.6.0.20
3ff87f37000-3ff87f3a000 rw-p 00136000 5e:05 6424618                      /data/gcc-4.9/lib64/libstdc++.so.6.0.20
3ff87f3a000-3ff87f4f000 rw-p 00000000 00:00 0
3ff87f80000-3ff87f88000 r-xp 00000000 5e:02 29842                        /usr/lib64/librt-2.17.so
3ff87f88000-3ff87f89000 r--p 00007000 5e:02 29842                        /usr/lib64/librt-2.17.so
3ff87f89000-3ff87f8a000 rw-p 00008000 5e:02 29842                        /usr/lib64/librt-2.17.so
3ff88000000-3ff88003000 r-xp 00000000 5e:02 5049                         /usr/lib64/libdl-2.17.so
3ff88003000-3ff88004000 r--p 00002000 5e:02 5049                         /usr/lib64/libdl-2.17.so
3ff88004000-3ff88005000 rw-p 00003000 5e:02 5049                         /usr/lib64/libdl-2.17.so
3ff88100000-3ff88123000 r-xp 00000000 5e:02 2626                         /usr/lib64/ld-2.17.so
3ff88123000-3ff88124000 r--p 00022000 5e:02 2626                         /usr/lib64/ld-2.17.so
3ff88124000-3ff88125000 rw-p 00023000 5e:02 2626                         /usr/lib64/ld-2.17.so
3ff88125000-3ff88126000 rw-p 00000000 00:00 0
3ff8816f000-3ff88170000 rw-p 00000000 00:00 0
3ff88170000-3ff88171000 ---p 00000000 00:00 0
3ff88171000-3ff8817e000 rw-p 00000000 00:00 0
3ff8817e000-3ff88180000 r-xp 00000000 00:00 0                            [vdso]
3fffc1df000-3fffc200000 rw-p 00000000 00:00 0                            [stack]

[apapirovski]

cc @nodejs/zlib @nodejs/async_hooks

[addaleax]

I think this is going to be nearly impossible to debug without having access to a machine on which this is reproducible. /cc @nodejs/build

Locally everything seems fine to me. As always with these kinds of issues, it would be great to have valgrind output (python tools/test.py --valgrind --repeat=100 -J async-hooks/test-zlib.zlib-binding.deflate) or core dumps for a crash available…

[apapirovski]

Yeah... I tried running my own machine with a similar setup but couldn't reproduce the crash or find anything with valgrind. Sounds like we will definitely need access.

[addaleax]

Maybe somebody like @Trott can ssh into one of these machines and run that command? That might be a huge help for finding a point to start. (Maybe first share the output privately though, if there is any, since memory corruption bugs sometimes can have security implications…)

[maclover7]

@apapirovski @addaleax What would be the best way for build to help debug this? To run valgrind and then share the output privately?

[addaleax]

@maclover7 That is at least worth the attempt, yes

[Trott]

I'm doing a bisect to find the commit that caused this.

https://ci.nodejs.org/job/node-test-commit-linux/19012/ shows that 4f0ab76 is good. So it's a commit that landed after that.

[maclover7]

Here's an interesting thing I found in Jenkins, not sure if helpful or not: https://ci.nodejs.org/job/node-test-commit-linuxone/1556/nodes=rhel72-s390x/testReport/(root)/test/async_hooks_test_zlib_zlib_binding_deflate/

[Trott]

According to @mcollina, this bug will be fixed with #19989.

[mcollina]

@Trott I think I have been misunderstood, I do not think this will be fixed by that update. I will test it.
It is very likely that it is: #20516.

[Trott]

OK, then, back to bisecting!

3654cd4 is good, so whatever caused this issue landed after that. https://ci.nodejs.org/job/node-test-commit-linux/19017/

[mcollina]

@Trott then that's definitely not the bug I'm talking about.