Segmentation fault (core dumped)

[luxinmu]

编译完成
[root@localhost phxqueue-master]# ll ~/phxqueue-master/bin
total 81388
-rwxr-x--- 1 root root 5481431 Sep 13 19:24 consumer_main
-rwxr-x--- 1 root root 6164740 Sep 13 19:24 lock_main
-rwxr-x--- 1 root root 3599574 Sep 13 19:24 lock_tool_main
-rwxr-x--- 1 root root 5117934 Sep 13 19:24 producer_benchmark_main
-rwxr-x--- 1 root root 5360373 Sep 13 19:24 scheduler_main
-rwxr-x--- 1 root root 3596118 Sep 13 19:24 scheduler_tool_main
-rwxr-x--- 1 root root 6207847 Sep 13 19:24 store_main
-rwxr-x--- 1 root root 3599327 Sep 13 19:24 store_tool_main
-rwxr-x--- 1 root root 3831632 Sep 13 19:24 test_config_main
-rwxr-x--- 1 root root 4260253 Sep 13 19:24 test_consumer_main
-rwxr-x--- 1 root root 4087214 Sep 13 19:24 test_load_config_main
-rwxr-x--- 1 root root 4937975 Sep 13 19:24 test_lock_main
-rwxr-x--- 1 root root 187347 Sep 13 19:24 test_log_main
-rwxr-x--- 1 root root 143758 Sep 13 19:24 test_notifierpool_main
-rwxr-x--- 1 root root 3677555 Sep 13 19:24 test_plugin_main
-rwxr-x--- 1 root root 5106439 Sep 13 19:24 test_producer_echo_main
-rwxr-x--- 1 root root 4076620 Sep 13 19:24 test_producer_main
-rwxr-x--- 1 root root 4872753 Sep 13 19:24 test_rpc_config_main
-rwxr-x--- 1 root root 4014373 Sep 13 19:24 test_scheduler_main
-rwxr-x--- 1 root root 4978414 Sep 13 19:24 test_store_main

运行时报错
[root@localhost phxqueue-master]# bin/store_main -c etc/store_server.0.conf
Segmentation fault (core dumped)

没有日志生成
[root@localhost log]# ls /root/phxqueue-master/log/*
/root/phxqueue-master/log/clear_log.sh
/root/phxqueue-master/log/consumer.0:
/root/phxqueue-master/log/consumer.1:
/root/phxqueue-master/log/consumer.2:
/root/phxqueue-master/log/lock.0:
/root/phxqueue-master/log/lock.1:
/root/phxqueue-master/log/lock.2:
/root/phxqueue-master/log/scheduler.0:
/root/phxqueue-master/log/scheduler.1:
/root/phxqueue-master/log/scheduler.2:
/root/phxqueue-master/log/store.0:
/root/phxqueue-master/log/store.1:
/root/phxqueue-master/log/store.2:

gdb查看core文件
[root@localhost phxqueue-master]# gdb bin/store_main core.2953
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-92.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /root/phxqueue-master/bin/store_main...done.
[New Thread 2953]
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /opt/gcc/gcc-4.8.2/lib64/libstdc++.so.6...done.
Loaded symbols for /opt/gcc/gcc-4.8.2/lib64/libstdc++.so.6
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /opt/gcc/gcc-4.8.2/lib64/libgcc_s.so.1...done.
Loaded symbols for /opt/gcc/gcc-4.8.2/lib64/libgcc_s.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Core was generated by bin/store_main -c etc/store_server.0.conf'. Program terminated with signal 11, Segmentation fault. #0 0x00007fdd0e537c9c in vfprintf () from /lib64/libc.so.6 warning: File "/opt/gcc/gcc-4.8.2/lib64/libstdc++.so.6.0.18-gdb.py" auto-loading has been declined by your auto-load safe-path' set to "/usr/share/gdb/auto-load:/usr/lib/debug:/usr/bin/mono-gdb.py".
To enable execution of this file add
add-auto-load-safe-path /opt/gcc/gcc-4.8.2/lib64/libstdc++.so.6.0.18-gdb.py
line to your configuration file "/root/.gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/root/.gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"
Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.209.el6_9.2.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0 0x00007fdd0e537c9c in vfprintf () from /lib64/libc.so.6
#1 0x00007fdd0e55f5c2 in vsnprintf () from /lib64/libc.so.6
#2 0x0000000000480b22 in phxqueue::plugin::LoggerGoogle::Log(int, char const*, __va_list_tag*) ()
#3 0x000000000048083d in phxqueue::comm::Logger::LogVerbose(char const*, ...) ()
#4 0x000000000048331f in phxpaxos::Logger::LogWarning(char const*, ...) ()
#5 0x000000000049b989 in phxpaxos::LogStore::RebuildIndex(phxpaxos::Database*, int&) ()
#6 0x000000000049bcbe in phxpaxos::LogStore::Init(std::basic_string<char, std::char_traits, std::allocator > const&, int, phxpaxos::Database*) ()
#7 0x0000000000497062 in phxpaxos::Database::Init(std::basic_string<char, std::char_traits, std::allocator > const&, int) ()
#8 0x0000000000499068 in phxpaxos::MultiDatabase::Init(std::basic_string<char, std::char_traits, std::allocator > const&, int) ()
#9 0x0000000000484832 in phxpaxos::PNode::InitLogStorage(phxpaxos::Options const&, phxpaxos::LogStorage*&) ()
#10 0x00000000004855f2 in phxpaxos::PNode::Init(phxpaxos::Options const&, phxpaxos::NetWork*&) ()
#11 0x00000000004813d8 in phxpaxos::Node::RunNode(phxpaxos::Options const&, phxpaxos::Node*&) ()
#12 0x00000000004223a4 in phxqueue::store::Store::PaxosInit() ()
#13 0x0000000000422b3b in phxqueue::store::Store::Init() ()
#14 0x000000000040b9de in main ()
(gdb)

[root@localhost lib64]# uname -a
Linux localhost 2.6.32-504.el6.x86_64 #1 SMP Tue Sep 16 01:56:35 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost lib64]# rpm -qa|grep glibc
glibc-common-2.12-1.209.el6_9.2.x86_64
glibc-devel-2.12-1.209.el6_9.2.x86_64
glibc-2.12-1.209.el6_9.2.x86_64
glibc-headers-2.12-1.209.el6_9.2.x86_64
glibc-devel-2.12-1.209.el6_9.2.i686
glibc-2.12-1.209.el6_9.2.i686

请帮忙分析一下原因 是c库函数版本的问题吗？

非常感谢~！

[shangmacun]

我也是： GDB 跟踪：
(gdb) set args -c /root/opensource/phxqueue/etc/store_server.0.conf
(gdb) r
Starting program: /root/opensource/phxqueue/bin/store_main -c /root/opensource/phxqueue/etc/store_server.0.conf
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Could not create logging file: No such file or directory
COULD NOT CREATE A LOGGINGFILE 20170914-050051.5153!Could not create logging file: No such file or directory
COULD NOT CREATE A LOGGINGFILE 20170914-050051.5153!Could not create logging file: No such file or directory
COULD NOT CREATE A LOGGINGFILE 20170914-050051.5153!ERR: MakeArgs ret -1
[Inferior 1 (process 5153) exited with code 0377]

[unixliang]

你好，我们查一下原因。

[tinyshu]

我的也是./bin/store_main -c ./etc/store_server.0.conf
Segmentation fault (core dumped)

[guyannanfei25]

+1

[unixliang]

Fixed。是一个日志组件引起的core，用临时措施先规避了。后面我们会完全修复这个bug。
Fixed. There is some issue in log component that causes the core, it has avoided by temporary measures. We'll fix this bug completely later.
1cc429a

[luxinmu]

重新编译成功后，执行程序报了如下错误
[root@localhost phxqueue-master]# bin/store_main -c etc/store_server.0.conf
server already started, 1 io threads 10 workers
listen succ, ip 127.0.0.1 port 5100

[root@localhost phxqueue-master]# bin/store_main -c etc/store_server.1.conf
server already started, 1 io threads 10 workers
listen succ, ip 127.0.0.1 port 5200

[root@localhost phxqueue-master]# bin/store_main -c etc/store_server.2.conf
store_server.2没有输出

[root@localhost phxqueue-master]# ps -ef|grep store_main
root 11765 2811 10 19:27 pts/1 00:01:44 bin/store_main -c etc/store_server.0.conf
root 12467 12420 10 19:28 pts/2 00:01:34 bin/store_main -c etc/store_server.1.conf
root 13298 12445 0 19:33 pts/3 00:00:00 bin/store_main -c etc/store_server.2.conf
root 13544 13415 0 19:43 pts/5 00:00:00 grep store_main

[root@localhost phxqueue-master]# bin/consumer_main -c etc/consumer_server.0.conf

[root@localhost phxqueue-master]# bin/test_producer_echo_main
produce echo "jrDe7nbYgZ" failed return 10105!

[root@localhost consumer.0]# tail -f test.ERROR
E0914 19:40:32.338951 13196 logger_google.cpp:79] ERR: PN8phxqueue8consumer8ConsumerE::ChildRun:499 DoLock ret -1 vpid 35
E0914 19:40:34.339504 13196 logger_google.cpp:79] ERR: P10LockClient::ProtoGetLockInfo:170 phxrpc Open err. ip 127.0.0.1 port 7100
E0914 19:40:34.349123 13196 logger_google.cpp:79] ERR: PN15phxqueue_phxrpc8consumer8ConsumerE::GetLockInfo:92 ProtoGetLockInfo ret -1
E0914 19:40:34.349733 13196 logger_google.cpp:79] ERR: P10LockClient::ProtoGetLockInfo:170 phxrpc Open err. ip 127.0.0.1 port 7200
E0914 19:40:34.350112 13196 logger_google.cpp:79] ERR: PN15phxqueue_phxrpc8consumer8ConsumerE::GetLockInfo:92 ProtoGetLockInfo ret -1
E0914 19:40:34.350487 13196 logger_google.cpp:79] ERR: P10LockClient::ProtoGetLockInfo:170 phxrpc Open err. ip 127.0.0.1 port 7300
E0914 19:40:34.350874 13196 logger_google.cpp:79] ERR: PN15phxqueue_phxrpc8consumer8ConsumerE::GetLockInfo:92 ProtoGetLockInfo ret -1
E0914 19:40:34.352484 13196 logger_google.cpp:79] ERR: PN8phxqueue8consumer13HeartBeatLockE::DoLock:724 ERR: GetLockInfo ret -1
E0914 19:40:34.353129 13196 logger_google.cpp:79] ERR: PN8phxqueue8consumer13HeartBeatLockE::Lock:785 ERR: DoLock ret -1 vpid 35

[root@localhost log]# tail -f store.0/test.ERROR
E0914 19:48:22.315274 14309 logger_google.cpp:79] ERR: PN8phxqueue5store16KeepMasterThreadE::KeepMaster:245 MASTERSTAT: ERR: Propose err. paxos_ret 404 paxos_group_id 15 buf.length 27
E0914 19:48:23.815553 14309 logger_google.cpp:79] ERR: PN8phxqueue5store16KeepMasterThreadE::KeepMaster:245 MASTERSTAT: ERR: Propose err. paxos_ret 404 paxos_group_id 18 buf.length 27
E0914 19:48:25.342494 14309 logger_google.cpp:79] ERR: PN8phxqueue5store16KeepMasterThreadE::KeepMaster:245 MASTERSTAT: ERR: Propose err. paxos_ret 404 paxos_group_id 21 buf.length 27
E0914 19:48:26.842792 14309 logger_google.cpp:79] ERR: PN8phxqueue5store16KeepMasterThreadE::KeepMaster:245 MASTERSTAT: ERR: Propose err. paxos_ret 404 paxos_group_id 24 buf.length 27
E0914 19:48:28.291803 14309 logger_google.cpp:79] ERR: PN8phxqueue5store16KeepMasterThreadE::KeepMaster:245 MASTERSTAT: ERR: Propose err. paxos_ret 404 paxos_group_id 27 buf.length 27
E0914 19:48:30.233376 14309 logger_google.cpp:79] ERR: PN8phxqueue5store16KeepMasterThreadE::KeepMaster:245 MASTERSTAT: ERR: Propose err. paxos_ret 404 paxos_group_id 30 buf.length 27
E0914 19:48:31.734184 14309 logger_google.cpp:79] ERR: PN8phxqueue5store16KeepMasterThreadE::KeepMaster:245 MASTERSTAT: ERR: Propose err. paxos_ret 404 paxos_group_id 33 buf.length 27
E0914 19:48:33.518514 14309 logger_google.cpp:79] ERR: PN8phxqueue5store16KeepMasterThreadE::KeepMaster:245 MASTERSTAT: ERR: Propose err. paxos_ret 404 paxos_group_id 36 buf.length 27
E0914 19:48:35.019265 14309 logger_google.cpp:79] ERR: PN8phxqueue5store16KeepMasterThreadE::KeepMaster:245 MASTERSTAT: ERR: Propose err. paxos_ret 404 paxos_group_id 39 buf.length 27
E0914 19:48:37.500082 14309 logger_google.cpp:79] ERR: PN8phxqueue5store16KeepMasterThreadE::KeepMaster:245 MASTERSTAT: ERR: Propose err. paxos_ret 404 paxos_group_id 42 buf.length 27

请帮忙查看，非常感谢~

[unixliang]

Set skip_lock = 1 in topicconfig.conf if not use Lock.
Set use_dynamic_scale = 0 in topicconfig.conf if not use Scheduler.

若未部署Lock和Schedluer，请参阅README如上修改这两项配置。
或者pull最新代码 5c1b8c8 ，最新代码的etc/topicconfig.conf 中已按如上修改。

[guyannanfei25]

能不能提供一个docker image的下载地址，pull国外的镜像老是超时 :-(

[wangxingge]

docker pull phxqueue/phxqueue
把镜像加速设置成阿里的加速地址。。

话说这个phxqueue的镜像真的是太大了。 1.42GB.

[guyannanfei25]

docker 启动的时候执行/usr/local/bin/docker-entrypoint.sh命令，导致想kill掉这些进程的时候出现好多僵尸进程kill不掉，这种方式不太好

root 8 1.4 0.0 0 0 ? Zs 09:50 1:05 [store_main]
root 10 1.4 0.0 0 0 ? Zs 09:50 1:06 [store_main]
root 12 0.0 0.0 0 0 ? Zs 09:50 0:00 [consumer_main]
root 14 0.0 0.0 0 0 ? Zs 09:50 0:00 [consumer_main]
root 16 0.0 0.0 0 0 ? Zs 09:50 0:00 [consumer_main]
root 18 1.2 0.0 0 0 ? Zs 09:50 0:55 [lock_main]
root 20 1.1 0.0 0 0 ? Zs 09:50 0:55 [lock_main]
root 22 1.1 0.0 0 0 ? Zs 09:50 0:55 [lock_main]
root 23 0.0 0.0 0 0 ? Z 09:50 0:00 [consumer_main]
root 24 0.0 0.0 0 0 ? Z 09:50 0:00 [consumer_main]
root 25 0.1 0.0 0 0 ? Z 09:50 0:05 [consumer_main]