Skip to content

[pg] Segfault on grafana start #5246

@rekby

Description

@rekby
Stacktrace
GRpc memory quota was set but disabled due to issues with grpc quoter, to enable it use EnableGRpcMemoryQuota option
AddressSanitizer:DEADLYSIGNAL
=================================================================
==9403==ERROR: AddressSanitizer: SEGV on unknown address (pc 0x7f3b367c531e bp 0x9eb1c626b344b985 sp 0xb795c626bc293779 T15)
==9403==The signal is caused by a READ memory access.
==9403==Hint: this fault was caused by a dereference of a high value address (see register values below).  Disassemble the provided pc to learn which register was used.
    #0 0x7f3b367c531e  setjmp/../sysdeps/x86_64/__longjmp.S:111

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV setjmp/../sysdeps/x86_64/__longjmp.S:111
Thread T15 (ydbd.User) created by T0 here:
warning: address range table at offset 0x0 has a premature terminator entry at offset 0x10
warning: address range table at offset 0x30 has a premature terminator entry at offset 0x40
warning: address range table at offset 0x60 has a premature terminator entry at offset 0x70
warning: address range table at offset 0x90 has a premature terminator entry at offset 0xa0
warning: address range table at offset 0xc0 has a premature terminator entry at offset 0xd0
warning: address range table at offset 0xac0 has a premature terminator entry at offset 0xad0
warning: address range table at offset 0xb20 has a premature terminator entry at offset 0xb30
warning: address range table at offset 0xb50 has a premature terminator entry at offset 0xb60
warning: address range table at offset 0xb80 has a premature terminator entry at offset 0xb90
warning: address range table at offset 0xbb0 has a premature terminator entry at offset 0xbc0
warning: address range table at offset 0xbe0 has a premature terminator entry at offset 0xbf0
warning: address range table at offset 0xc10 has a premature terminator entry at offset 0xc20
warning: address range table at offset 0xc40 has a premature terminator entry at offset 0xc50
warning: address range table at offset 0xc70 has a premature terminator entry at offset 0xc80
warning: address range table at offset 0xca0 has a premature terminator entry at offset 0xcb0
warning: address range table at offset 0xcd0 has a premature terminator entry at offset 0xce0
warning: address range table at offset 0xd00 has a premature terminator entry at offset 0xd10
warning: address range table at offset 0xd30 has a premature terminator entry at offset 0xd40
warning: address range table at offset 0xd60 has a premature terminator entry at offset 0xd70
warning: address range table at offset 0xd90 has a premature terminator entry at offset 0xda0
warning: address range table at offset 0xdc0 has a premature terminator entry at offset 0xdd0
warning: address range table at offset 0xdf0 has a premature terminator entry at offset 0xe00
warning: address range table at offset 0xe20 has a premature terminator entry at offset 0xe30
warning: address range table at offset 0xe50 has a premature terminator entry at offset 0xe60
warning: address range table at offset 0xe80 has a premature terminator entry at offset 0xe90
warning: address range table at offset 0x13c0 has a premature terminator entry at offset 0x13d0
warning: address range table at offset 0x13f0 has a premature terminator entry at offset 0x1400
warning: address range table at offset 0x1420 has a premature terminator entry at offset 0x1430
warning: address range table at offset 0x1450 has a premature terminator entry at offset 0x1460
warning: address range table at offset 0x1570 has a premature terminator entry at offset 0x1580
warning: address range table at offset 0x15a0 has a premature terminator entry at offset 0x15b0
warning: address range table at offset 0x15d0 has a premature terminator entry at offset 0x15e0
warning: address range table at offset 0x1600 has a premature terminator entry at offset 0x1610
warning: address range table at offset 0x1630 has a premature terminator entry at offset 0x1640
warning: address range table at offset 0x1660 has a premature terminator entry at offset 0x1670
    #0 0x2008422c in __interceptor_pthread_create /home/rekby/ydbwork/ydb/contrib/libs/clang16-rt/lib/asan/asan_interceptors.cpp:208:3
    #1 0x202bc533 in (anonymous namespace)::TPosixThread::Start() /home/rekby/ydbwork/ydb/util/system/thread.cpp:229:27
    #2 0x202bc533 in TThread::Start() /home/rekby/ydbwork/ydb/util/system/thread.cpp:314:34
    #3 0x21e7fc83 in NActors::TBasicExecutorPool::Start() /home/rekby/ydbwork/ydb/ydb/library/actors/core/executor_pool_basic.cpp:466:32
    #4 0x21e661b5 in NActors::TCpuManager::Start() /home/rekby/ydbwork/ydb/ydb/library/actors/core/cpu_manager.cpp:81:32
    #5 0x21e5c4e4 in NActors::TActorSystem::Start() /home/rekby/ydbwork/ydb/ydb/library/actors/core/actorsystem.cpp:300:21
    #6 0x33e75e2e in NKikimr::TKikimrRunner::KikimrStart() /home/rekby/ydbwork/ydb/ydb/core/driver_lib/run/run.cpp:1669:22
    #7 0x334d1a30 in NKikimr::MainRun(NKikimr::TKikimrRunConfig const&, std::__y1::shared_ptr<NKikimr::TModuleFactories>) /home/rekby/ydbwork/ydb/ydb/core/driver_lib/run/main.cpp:46:17
    #8 0x33630dd1 in NKikimr::NDriverClient::TClientCommandServer::Run(NYdb::NConsoleClient::TClientCommand::TConfig&) /home/rekby/ydbwork/ydb/ydb/core/driver_lib/cli_utils/cli_cmds_server.cpp:51:12
    #9 0x3352ca1e in NYdb::NConsoleClient::TClientCommandTree::Run(NYdb::NConsoleClient::TClientCommand::TConfig&) /home/rekby/ydbwork/ydb/ydb/public/lib/ydb_cli/common/command.cpp:374:33
    #10 0x335200d0 in NKikimr::NDriverClient::NewClient(int, char**, std::__y1::shared_ptr<NKikimr::TModuleFactories>) /home/rekby/ydbwork/ydb/ydb/core/driver_lib/cli_utils/cli_cmds_root.cpp:82:26
    #11 0x334d3082 in NKikimr::Main(int, char**, std::__y1::shared_ptr<NKikimr::TModuleFactories>) /home/rekby/ydbwork/ydb/ydb/core/driver_lib/run/main.cpp:148:20
    #12 0x334d47cb in ParameterizedMain(int, char**, std::__y1::shared_ptr<NKikimr::TModuleFactories>) /home/rekby/ydbwork/ydb/ydb/core/driver_lib/run/main.cpp:198:16
    #13 0x2002930d in main /home/rekby/ydbwork/ydb/ydb/apps/ydbd/main.cpp:24:12
    #14 0x7f3b367acd8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

==9403==ABORTING

Reproduce with grafana:

  1. Run local YDB https://github.com/ydb-platform/ydb/wiki/Local-run-postgres-tests, with export ASAN_SYMBOLIZER_PATH=$(ya tool llvm-symbolizer --print-path)
  2. Clone https://github.com/ydb-platform/postgres-compatibility-tests (commit 3c21c218f087c942908d1d62d06edfc3e5c1f044)
  3. From root of repository run:
cat ./manual-in-progress/grafana/ydb-schema.sql | psql postgres://root:1234@localhost:5432/local
cat ./manual-in-progress/grafana/ydb-data.sql | psql postgres://root:1234@localhost:5432/local
docker-compose -f ./manual-in-progress/grafana/docker-compose.yaml down -vt 1 && docker-compose -f ./manual-in-progress/grafana/docker-compose.yaml up

After docker-compose command ydbd fails without backtrace within coredump.

I can see the stacktrace if (flacky):

  1. Run ydbd with ASAN_SYMBOLIZER_PATH
  2. Attach with gdb to the process: sudo ./ya tool gdb ~/ydbwork/ydb/ydb/apps/ydbd/ydbd --pid YDBD_PID, then execue continue.
  3. Run docker-compose -f ./manual-in-progress/grafana/docker-compose.yaml down -vt 1 && docker-compose -f ./manual-in-progress/grafana/docker-compose.yaml up for ydbd fails
  4. exit from gdb with detach from the process
  5. Restart ydbd through ydb_local

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions