flake in IpVersions/GolangIntegrationTest.LocalReply_DecodeHeader/IPv6

[alyssawilk]

[ RUN ] IpVersions/GolangIntegrationTest.LocalReply_DecodeHeader/IPv6
[2023-03-20 15:15:54.964][15][critical][backtrace] [./source/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x745425a1590
[2023-03-20 15:15:54.964][15][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
[2023-03-20 15:15:54.964][15][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 0/1.26.0-dev/test/RELEASE/BoringSSL
[2023-03-20 15:15:54.965][15][critical][backtrace] [./source/server/backtrace.h:96] #0: runtime.sigfwd.abi0 [0x7f378d95a77d]
[2023-03-20 15:15:54.965][15][critical][backtrace] [./source/server/backtrace.h:96] #1: runtime.sigfwdgo [0x7f378d940054]
[2023-03-20 15:15:54.965][15][critical][backtrace] [./source/server/backtrace.h:96] #2: runtime.sigtrampgo [0x7f378d93e7e7]
[2023-03-20 15:15:54.966][15][critical][backtrace] [./source/server/backtrace.h:96] #3: runtime.sigtrampgo.abi0 [0x7f378d95b390]
[2023-03-20 15:15:54.966][15][critical][backtrace] [./source/server/backtrace.h:96] #4: runtime.sigtramp.abi0 [0x7f378d95a7dd]
[2023-03-20 15:15:54.966][15][critical][backtrace] [./source/server/backtrace.h:96] #5: runtime.sigfwd.abi0 [0x7f378ea8617d]
[2023-03-20 15:15:54.966][15][critical][backtrace] [./source/server/backtrace.h:96] #6: runtime.sigfwdgo [0x7f378ea6bad4]
[2023-03-20 15:15:54.966][15][critical][backtrace] [./source/server/backtrace.h:96] #7: runtime.sigtrampgo [0x7f378ea6a267]
[2023-03-20 15:15:54.967][15][critical][backtrace] [./source/server/backtrace.h:96] #8: runtime.sigtrampgo.abi0 [0x7f378ea86d90]
[2023-03-20 15:15:54.967][15][critical][backtrace] [./source/server/backtrace.h:96] #9: runtime.sigtramp.abi0 [0x7f378ea861dd]
[2023-03-20 15:15:54.967][15][critical][backtrace] [./source/server/backtrace.h:96] #10: runtime.sigfwd.abi0 [0x7f37658c6e5d]
[2023-03-20 15:15:54.967][15][critical][backtrace] [./source/server/backtrace.h:96] #11: runtime.sigfwdgo [0x7f37658ac554]
[2023-03-20 15:15:54.967][15][critical][backtrace] [./source/server/backtrace.h:96] #12: runtime.sigtrampgo [0x7f37658aace7]
[2023-03-20 15:15:54.968][15][critical][backtrace] [./source/server/backtrace.h:96] #13: runtime.sigtrampgo.abi0 [0x7f37658c7a70]
[2023-03-20 15:15:54.968][15][critical][backtrace] [./source/server/backtrace.h:96] #14: runtime.sigtramp.abi0 [0x7f37658c6ebd]
[2023-03-20 15:15:54.968][15][critical][backtrace] [./source/server/backtrace.h:96] #15: __restore_rt [0x7f3792f90420]
[2023-03-20 15:15:54.977][15][critical][backtrace] [./source/server/backtrace.h:96] #16: Envoy::IntegrationCodecClient::sendData() [0x2c7fb78]
[2023-03-20 15:15:54.987][15][critical][backtrace] [./source/server/backtrace.h:96] #17: Envoy::GolangIntegrationTest::testSendLocalReply() [0x1484085]
[2023-03-20 15:15:54.993][15][critical][backtrace] [./source/server/backtrace.h:96] #18: Envoy::GolangIntegrationTest_LocalReply_DecodeHeader_Test::TestBody() [0x1483d92]
[2023-03-20 15:15:55.001][15][critical][backtrace] [./source/server/backtrace.h:96] #19: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x3e53f1c]
[2023-03-20 15:15:55.008][15][critical][backtrace] [./source/server/backtrace.h:96] #20: testing::Test::Run() [0x3e53ddd]
[2023-03-20 15:15:55.015][15][critical][backtrace] [./source/server/backtrace.h:96] #21: testing::TestInfo::Run() [0x3e54ff7]
[2023-03-20 15:15:55.021][15][critical][backtrace] [./source/server/backtrace.h:96] #22: testing::TestSuite::Run() [0x3e55db9]
[2023-03-20 15:15:55.027][15][critical][backtrace] [./source/server/backtrace.h:96] #23: testing::internal::UnitTestImpl::RunAllTests() [0x3e632ef]
[2023-03-20 15:15:55.034][15][critical][backtrace] [./source/server/backtrace.h:96] #24: testing::internal::HandleExceptionsInMethodIfSupported<>() [0x3e62d5c]
[2023-03-20 15:15:55.040][15][critical][backtrace] [./source/server/backtrace.h:96] #25: testing::UnitTest::Run() [0x3e62bdf]
[2023-03-20 15:15:55.046][15][critical][backtrace] [./source/server/backtrace.h:96] #26: Envoy::TestRunner::RunTests() [0x371e493]
[2023-03-20 15:15:55.053][15][critical][backtrace] [./source/server/backtrace.h:96] #27: main [0x371cf1e]
[2023-03-20 15:15:55.053][15][critical][backtrace] [./source/server/backtrace.h:96] #28: __libc_start_main [0x7f3792dae083]

[doujiang24]

Thanks @alyssawilk , could you please provide more details?
i.e. what's the system environment to reproduce this failure? Does it always reproducible or just fail randomly?

I can't reproduce it on my side:

[ RUN      ] IpVersions/GolangIntegrationTest.LocalReply_DecodeHeader/IPv6
[external/com_google_absl/absl/flags/internal/flag.cc : 113] RAW: Restore saved value of envoy_reloadable_features_runtime_initialized to: false
[external/com_google_absl/absl/flags/internal/flag.cc : 113] RAW: Restore saved value of envoy_reloadable_features_no_extension_lookup_by_name to: true
[       OK ] IpVersions/GolangIntegrationTest.LocalReply_DecodeHeader/IPv6 (303 ms)

system environment on my side:

Ubuntu 22.04
Intel Xeon CPU
--config=gcc
gcc version 11.3.0

Also, does this failure only happens on IPv6?
FYI, there is no special handle for IPv6 in the golang extension. Thanks!

[alyssawilk]

This was postsubmit CI

https://dev.azure.com/cncf/envoy/_build/results?buildId=131502&view=logs&j=8c169225-0ae8-53bd-947f-07cb81846cb5&t=427034ac-ea02-50a9-4444-aee028d5fd94&l=264

I suggest trying to run with --runs_per_test=1000 or trying the general instructions in test/integration/README.md to see if you can reproduce.

[alyssawilk]

also https://dev.azure.com/cncf/envoy/_build/results?buildId=131565&view=logs&j=767be981-567e-57d8-68c3-2140ede0a0bd&t=8a34f9ed-4946-5fcf-7cd6-99c93070884d&l=537

I'm going to disable the test until you have a chance to take a look

2023-03-20T23:36:58.6081702Z [ RUN ] IpVersions/GolangIntegrationTest.LuaRespondAfterGoHeaderContinue/IPv6
2023-03-20T23:36:58.6083638Z [2023-03-20 23:36:58.280][401006][critical][backtrace] [./source/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x250ec0be13a4
2023-03-20T23:36:58.6085404Z [2023-03-20 23:36:58.281][401006][critical][backtrace] [./source/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
2023-03-20T23:36:58.6087120Z [2023-03-20 23:36:58.281][401006][critical][backtrace] [./source/server/backtrace.h:92] Envoy version: 0/1.26.0-dev/test/RELEASE/BoringSSL
2023-03-20T23:36:58.6088756Z [2023-03-20 23:36:58.281][401006][critical][backtrace] [./source/server/backtrace.h:96] #0: runtime.sigfwd.abi0 [0xffff865737c0]

[doujiang24]

okay, thanks, i'm working on it, need some time to figure out.