Skip to content

stream load leads BE fatal error #3877

Closed
@rickif

Description

The bthread::ExecutionQueue used in StarRocks may get fatal error when the number of tasks in the queue is overloading.

Steps to reproduce the behavior (Required)

  1. creating table with massive buckets
    To make it easier to reproduce, I create a table with 2000 buckets.
  2. making stream load to the table created before
    To make it easier to reproduce, I make stream load with 32 parallelism and 10MB batch.

Expected behavior (Required)

The BE runs properly.

Real behavior (Required)

The BE get fatal error.

W0305 14:53:20.082063 19490 async_delta_writer_executor.h:32] Service unavailable: Thread pool is at capacity (16/16 tasks running, 40960/40960 tasks queued)
F0305 14:53:20.082093 19490 execution_queue.cpp:120] Fail to submit task: Resource temporarily unavailable [11]
*** Check failure stack trace: ***
    @          0x3b2f86d  google::LogMessage::Fail()
    @          0x3b31b49  google::LogMessage::SendToLog()
    @          0x3b2f3e9  google::LogMessage::Flush()
    @          0x3b2f5f2  google::LogMessage::~LogMessage()
    @          0x3c0c01c  bthread::ExecutionQueueBase::start_execute()
    @          0x30cf32f  starrocks::vectorized::AsyncDeltaWriter::write()
    @          0x1e2acec  starrocks::TabletsChannel::add_chunk()
    @          0x1e27565  starrocks::LoadChannel::add_chunk()
    @          0x1e2405b  starrocks::LoadChannelMgr::add_chunk()
    @          0x3c5f4fe  brpc::policy::ProcessRpcRequest()
    @          0x3c55f07  brpc::ProcessInputMessage()
    @          0x3c56db3  brpc::InputMessenger::OnNewMessages()
    @          0x3cfde2e  brpc::Socket::ProcessEvent()
    @          0x3da2c6f  bthread::TaskGroup::task_runner()
    @          0x3d97191  bthread_make_fcontext

StarRocks version (Required)

main, branch-2.1

Metadata

Assignees

No one assigned

    Labels

    type/bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions