Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

grpc internal abort in pressure test #5013

Open
LittleFall opened this issue May 27, 2022 · 0 comments
Open

grpc internal abort in pressure test #5013

LittleFall opened this issue May 27, 2022 · 0 comments

Comments

@LittleFall
Copy link
Contributor

LittleFall commented May 27, 2022

Steps that may reproduce the issue

  1. Prepare TPCH database, SF=10, tiflash replicas=2
  2. Don't analyze any tables before running queries.
  3. Parrellely run 20 TPCH queries, each one runs queries sequentially.

"grpc:/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/tics/contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc, line number: 687, log msg : server stream 1025 still included in list 1"

for (int i = 0; i < STREAM_LIST_COUNT; i++) {
    if (GPR_UNLIKELY(included[i])) {
      gpr_log(GPR_ERROR, "%s stream %d still included in list %d",
              t->is_client ? "client" : "server", id, i);
      abort();
    }
  }

stack

[2022/05/24 22:16:15.762 +08:00] [ERROR] [BaseDaemon.cpp:570] ["BaseDaemon:
       0x1e88ca1    faultSignalHandler(int, siginfo_t*, void*) [tiflash+32017569]
                    libs/libdaemon/src/BaseDaemon.cpp:221
  0x7f7547b68630    <unknown symbol> [libpthread.so.0+63024]
  0x7f75475ab387    raise [libc.so.6+222087]
  0x7f75475aca78    abort [libc.so.6+227960]
       0x7fa692d    destroy_stream_locked(void*, grpc_error*) [tiflash+133851437]
                    contrib/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:726
       0x7e484fa    grpc_combiner_continue_exec_ctx() [tiflash+132416762]
                    contrib/grpc/src/core/lib/iomgr/combiner.cc:236
       0x7e50695    pollset_work(grpc_pollset*, grpc_pollset_worker**, long) [tiflash+132449941]
                    contrib/grpc/src/core/lib/iomgr/ev_epollex_linux.cc:1136
       0x7e83b4d    cq_next(grpc_completion_queue*, gpr_timespec, void*) [tiflash+132660045]
                    contrib/grpc/src/core/lib/surface/completion_queue.cc:1038
       0x1d26bc3    DB::handleRpcs(grpc_impl::ServerCompletionQueue*, Poco::Logger*) [tiflash+30567363]
                    dbms/src/Server/Server.cpp:528
       0x1d74375    auto DB::wrapInvocable<std::__1::function<void ()> >(bool, std::__1::function<void ()>&&)::'lambda'()::operator()() [tiflash+30884725]
                    dbms/src/Common/wrapInvocable.h:36
       0x1d74665    std::__1::packaged_task<void ()>::operator()() [tiflash+30885477]
                    /usr/local/bin/../include/c++/v1/future:2089
       0x1d77d23    DB::DynamicThreadPool::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&) [tiflash+30899491]
                    dbms/src/Common/DynamicThreadPool.cpp:101
       0x1d77380    DB::DynamicThreadPool::fixedWork(unsigned long) [tiflash+30897024]
                    dbms/src/Common/DynamicThreadPool.cpp:115
       0x1d784eb    void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*)(unsigned long), DB::DynamicThreadPool*, unsigned long&>(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&)::'lambda'(auto&&...), DB::DynamicThreadPool*, unsigned long> >(void*) [tiflash+30901483]
                    /usr/local/bin/../include/c++/v1/thread:291
  0x7f7547b60ea5    start_thread [libpthread.so.0+32421]"] [thread_id=755]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment