Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DocDB] CppCassandraDriverTest.TestTableBackfillWithLeaderMoves flaky in master #10753

Closed
bmatican opened this issue Dec 1, 2021 · 3 comments
Assignees
Labels
area/docdb YugabyteDB core features kind/bug This issue is a bug kind/failing-test Tests and testing infra priority/high High Priority

Comments

@bmatican
Copy link
Contributor

bmatican commented Dec 1, 2021

Jira Link: DB-997

Description

Seen it recently failing in >= 3 build types

https://detective-gcp.dev.yugabyte.com/stability/test?branch=master&class=CppCassandraDriverTest&name=TestTableBackfillWithLeaderMoves

@bmatican bmatican added kind/failing-test Tests and testing infra area/docdb YugabyteDB core features labels Dec 1, 2021
@yugabyte-ci yugabyte-ci added kind/bug This issue is a bug priority/medium Medium priority issue labels Jun 8, 2022
@bmatican
Copy link
Contributor Author

Seems there's a race on setting the cluster config vs the LB accessing it. cc @sanketkedia

[m-1] WARNING: ThreadSanitizer: data race (pid=23969)
[m-1]   Write of size 8 at 0x7b0800109560 by thread T40 (mutexes: read M2822, write M92177918092252576):
[m-1]     #0 operator delete(void*) /opt/yb-build/llvm/yb-llvm-v12.0.1-yb-1-1651816038-bdb147e6-centos7-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_new_delete.cpp:126:3 (yb-master+0x2f76be)
[m-1]     #1 google::protobuf::internal::ArenaStringPtr::DestroyNoArena(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/include/google/protobuf/arenastring.h:264:7 (liblog_proto.so+0x1fa4a)
[m-1]     #2 yb::HostPortPB::SharedDtor() ${BUILD_ROOT}/src/yb/common/common_net.pb.cc:239:9 (libyb_common_proto.so+0x36fa15)
[m-1]     #3 yb::HostPortPB::~HostPortPB() ${BUILD_ROOT}/src/yb/common/common_net.pb.cc:235:3 (libyb_common_proto.so+0x36f9b3)
[m-1]     #4 yb::HostPortPB::~HostPortPB() ${BUILD_ROOT}/src/yb/common/common_net.pb.cc:233:27 (libyb_common_proto.so+0x36fa59)
[m-1]     #5 google::protobuf::internal::GenericTypeHandler<yb::HostPortPB>::Delete(yb::HostPortPB*, google::protobuf::Arena*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/include/google/protobuf/repeated_field.h:623:7 (libmaster.so+0x99e170)
[m-1]     #6 void google::protobuf::internal::RepeatedPtrFieldBase::Destroy<google::protobuf::RepeatedPtrField<yb::HostPortPB>::TypeHandler>() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/include/google/protobuf/repeated_field.h:1473:7 (libmaster.so+0x99e0f9)
[m-1]     #7 google::protobuf::RepeatedPtrField<yb::HostPortPB>::~RepeatedPtrField() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/include/google/protobuf/repeated_field.h:1934:3 (libmaster.so+0x936b8a)
[m-1]     #8 yb::master::BlacklistPB::~BlacklistPB() ${BUILD_ROOT}/src/yb/master/catalog_entity_info.pb.cc:7020:1 (libmaster_proto.so+0x90de94)
[m-1]     #9 yb::master::BlacklistPB::~BlacklistPB() ${BUILD_ROOT}/src/yb/master/catalog_entity_info.pb.cc:7017:29 (libmaster_proto.so+0x90dee9)
[m-1]     #10 yb::master::SysClusterConfigEntryPB::SharedDtor() ${BUILD_ROOT}/src/yb/master/catalog_entity_info.pb.cc:7868:44 (libmaster_proto.so+0x910464)
[m-1]     #11 yb::master::SysClusterConfigEntryPB::~SysClusterConfigEntryPB() ${BUILD_ROOT}/src/yb/master/catalog_entity_info.pb.cc:7859:3 (libmaster_proto.so+0x910323)
[m-1]     #12 yb::master::Persistent<yb::master::SysClusterConfigEntryPB, (yb::master::SysRowEntryType)3>::~Persistent() ${BUILD_ROOT}/../../ent/src/yb/master/../../../../src/yb/master/catalog_entity_info.h:109:8 (libmaster.so+0x99d069)
[m-1]     #13 std::__1::default_delete<yb::master::PersistentClusterConfigInfo>::operator()(yb::master::PersistentClusterConfigInfo*) const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/memory:1423:5 (libmaster.so+0x99d16e)
[m-1]     #14 std::__1::unique_ptr<yb::master::PersistentClusterConfigInfo, std::__1::default_delete<yb::master::PersistentClusterConfigInfo> >::reset(yb::master::PersistentClusterConfigInfo*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/memory:1684:7 (libmaster.so+0x99d0dd)
[m-1]     #15 yb::CowObject<yb::master::PersistentClusterConfigInfo>::CommitMutation() ${BUILD_ROOT}/../../src/yb/util/cow_object.h:100:18 (libmaster.so+0x99ddab)
[m-1]     #16 yb::CowWriteLock<yb::master::PersistentClusterConfigInfo>::Commit() ${BUILD_ROOT}/../../src/yb/util/cow_object.h:250:11 (libmaster.so+0x936a31)
[m-1]     #17 yb::master::CatalogManager::SetClusterConfig(yb::master::ChangeMasterClusterConfigRequestPB const*, yb::master::ChangeMasterClusterConfigResponsePB*) ${BUILD_ROOT}/../../src/yb/master/catalog_manager.cc:10722:5 (libmaster.so+0x9182f5)
[m-1]     #18 void yb::master::MasterServiceBase::HandleIn<yb::master::CatalogManager, yb::master::ChangeMasterClusterConfigRequestPB, yb::master::ChangeMasterClusterConfigResponsePB>(yb::master::ChangeMasterClusterConfigRequestPB const*, yb::master::ChangeMasterClusterConfigResponsePB*, yb::rpc::RpcContext*, yb::Status (yb::master::CatalogManager::*)(yb::master::ChangeMasterClusterConfigRequestPB const*, yb::master::ChangeMasterClusterConfigResponsePB*), char const*, int, char const*, yb::StronglyTypedBool<yb::master::HoldCatalogLock_Tag>)::'lambda'()::operator()() const ${BUILD_ROOT}/../../src/yb/master/master_service_base-internal.h:131:14 (libmaster.so+0xb0afe3)
[m-1]     #19 void yb::master::MasterServiceBase::HandleOnLeader<yb::master::ChangeMasterClusterConfigRequestPB, yb::master::ChangeMasterClusterConfigResponsePB, void yb::master::MasterServiceBase::HandleIn<yb::master::CatalogManager, yb::master::ChangeMasterClusterConfigRequestPB, yb::master::ChangeMasterClusterConfigResponsePB>(yb::master::ChangeMasterClusterConfigRequestPB const*, yb::master::ChangeMasterClusterConfigResponsePB*, yb::rpc::RpcContext*, yb::Status (yb::master::CatalogManager::*)(yb::master::ChangeMasterClusterConfigRequestPB const*, yb::master::ChangeMasterClusterConfigResponsePB*), char const*, int, char const*, yb::StronglyTypedBool<yb::master::HoldCatalogLock_Tag>)::'lambda'()>(yb::master::CatalogManager const*, yb::master::ChangeMasterClusterConfigRequestPB*, yb::rpc::RpcContext*, yb::master::ChangeMasterClusterConfigResponsePB, char const*, int, char const*, yb::StronglyTypedBool<yb::master::HoldCatalogLock_Tag>) ${BUILD_ROOT}/../../src/yb/master/master_service_base-internal.h:84:20 (libmaster.so+0xb0ae86)
[m-1]     #20 void yb::master::MasterServiceBase::HandleIn<yb::master::CatalogManager, yb::master::ChangeMasterClusterConfigRequestPB, yb::master::ChangeMasterClusterConfigResponsePB>(yb::master::ChangeMasterClusterConfigRequestPB const*, yb::master::ChangeMasterClusterConfigResponsePB*, yb::rpc::RpcContext*, yb::Status (yb::master::CatalogManager::*)(yb::master::ChangeMasterClusterConfigRequestPB const*, yb::master::ChangeMasterClusterConfigResponsePB*), char const*, int, char const*, yb::StronglyTypedBool<yb::master::HoldCatalogLock_Tag>) ${BUILD_ROOT}/../../src/yb/master/master_service_base-internal.h:130:3 (libmaster.so+0xb0ad6e)
[m-1]     #21 yb::master::(anonymous namespace)::MasterClusterServiceImpl::ChangeMasterClusterConfig(yb::master::ChangeMasterClusterConfigRequestPB const*, yb::master::ChangeMasterClusterConfigResponsePB*, yb::rpc::RpcContext) ${BUILD_ROOT}/../../src/yb/master/master_cluster_service.cc:264:5 (libmaster.so+0xb0617f)
[m-1]     #22 yb::master::MasterClusterIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_12::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::master::ChangeMasterClusterConfigRequestPB const*, yb::master::ChangeMasterClusterConfigResponsePB*, yb::rpc::RpcContext)::operator()(yb::master::ChangeMasterClusterConfigRequestPB const*, yb::master::ChangeMasterClusterConfigResponsePB*, yb::rpc::RpcContext) const ${BUILD_ROOT}/src/yb/master/master_cluster.service.cc:588:9 (libmaster_proto.so+0x6ef722)
[m-1]     #23 auto yb::rpc::HandleCall<yb::rpc::RpcCallPBParamsImpl<yb::master::ChangeMasterClusterConfigRequestPB, yb::master::ChangeMasterClusterConfigResponsePB>, yb::master::MasterClusterIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_12::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::master::ChangeMasterClusterConfigRequestPB const*, yb::master::ChangeMasterClusterConfigResponsePB*, yb::rpc::RpcContext)>(std::__1::shared_ptr<yb::rpc::InboundCall>, yb::master::MasterClusterIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_12::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::master::ChangeMasterClusterConfigRequestPB const*, yb::master::ChangeMasterClusterConfigResponsePB*, yb::rpc::RpcContext)) ${BUILD_ROOT}/../../src/yb/rpc/local_call.h:124:7 (libmaster_proto.so+0x6ef5d3)
[m-1]     #24 yb::master::MasterClusterIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_12::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const ${BUILD_ROOT}/src/yb/master/master_cluster.service.cc:586:7 (libmaster_proto.so+0x6ef35f)
[m-1]     #25 decltype(std::__1::forward<yb::master::MasterClusterIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_12&>(fp)(std::__1::forward<std::__1::shared_ptr<yb::rpc::InboundCall> >(fp0))) std::__1::__invoke<yb::master::MasterClusterIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_12&, std::__1::shared_ptr<yb::rpc::InboundCall> >(yb::master::MasterClusterIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_12&, std::__1::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/type_traits:3694:1 (libmaster_proto.so+0x6ef2bb)
[m-1]     #26 void std::__1::__invoke_void_return_wrapper<void, true>::__call<yb::master::MasterClusterIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_12&, std::__1::shared_ptr<yb::rpc::InboundCall> >(yb::master::MasterClusterIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_12&, std::__1::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/__functional_base:348:9 (libmaster_proto.so+0x6ef228)
[m-1]     #27 std::__1::__function::__alloc_func<yb::master::MasterClusterIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_12, std::__1::allocator<yb::master::MasterClusterIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_12>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:1558:16 (libmaster_proto.so+0x6ef1e3)
[m-1]     #28 std::__1::__function::__func<yb::master::MasterClusterIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_12, std::__1::allocator<yb::master::MasterClusterIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_12>, void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:1732:12 (libmaster_proto.so+0x6ee47c)
[m-1]     #29 std::__1::__function::__value_func<void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>&&) const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:1885:16 (libcdc_service_proto.so+0x1202b6)
[m-1]     #30 std::__1::function<void (std::__1::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::__1::shared_ptr<yb::rpc::InboundCall>) const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:2560:12 (libcdc_service_proto.so+0x11bda8)
[m-1]     #31 yb::master::MasterClusterIf::Handle(std::__1::shared_ptr<yb::rpc::InboundCall>) ${BUILD_ROOT}/src/yb/master/master_cluster.service.cc:391:3 (libmaster_proto.so+0x6dd45a)
[m-1]     #32 yb::rpc::ServicePoolImpl::Handle(std::__1::shared_ptr<yb::rpc::InboundCall>) ${BUILD_ROOT}/../../src/yb/rpc/service_pool.cc:270:19 (libyrpc.so+0x3cccfd)
[m-1]     #33 yb::rpc::InboundCall::InboundCallTask::Run() ${BUILD_ROOT}/../../src/yb/rpc/inbound_call.cc:237:13 (libyrpc.so+0x2dcc06)
[m-1]     #34 yb::rpc::(anonymous namespace)::Worker::Execute() ${BUILD_ROOT}/../../src/yb/rpc/thread_pool.cc:104:15 (libyrpc.so+0x3ea7fc)
[m-1]     #35 decltype(*(std::__1::forward<yb::rpc::(anonymous namespace)::Worker*&>(fp0)).*fp()) std::__1::__invoke<void (yb::rpc::(anonymous namespace)::Worker::*&)(), yb::rpc::(anonymous namespace)::Worker*&, void>(void (yb::rpc::(anonymous namespace)::Worker::*&)(), yb::rpc::(anonymous namespace)::Worker*&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/type_traits:3635:1 (libyrpc.so+0x3ebbd3)
[m-1]     #36 std::__1::__bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::__1::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::__1::tuple<>, __is_valid_bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::__1::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::__1::tuple<yb::rpc::(anonymous namespace)::Worker*>, 0ul, std::__1::tuple<> >(void (yb::rpc::(anonymous namespace)::Worker::*&)(), std::__1::tuple<yb::rpc::(anonymous namespace)::Worker*>&, std::__1::__tuple_indices<0ul>, std::__1::tuple<>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:2857:12 (libyrpc.so+0x3ebb48)
[m-1]     #37 std::__1::__bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::__1::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::__1::tuple<>, __is_valid_bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::__1::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::__1::tuple<> >::value>::type std::__1::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>::operator()<>() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:2890:20 (libyrpc.so+0x3ebafd)
[m-1]     #38 decltype(std::__1::forward<std::__1::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>(fp)()) std::__1::__invoke<std::__1::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>(std::__1::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/type_traits:3694:1 (libyrpc.so+0x3ebab9)
[m-1]     #39 void std::__1::__invoke_void_return_wrapper<void, true>::__call<std::__1::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>(std::__1::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/__functional_base:348:9 (libyrpc.so+0x3eba49)
[m-1]     #40 std::__1::__function::__alloc_func<std::__1::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>, std::__1::allocator<std::__1::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&> >, void ()>::operator()() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:1558:16 (libyrpc.so+0x3eba11)
[m-1]     #41 std::__1::__function::__func<std::__1::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>, std::__1::allocator<std::__1::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&> >, void ()>::operator()() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:1732:12 (libyrpc.so+0x3eacbd)
[m-1]     #42 std::__1::__function::__value_func<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:1885:16 (libmaster.so+0xd09084)
[m-1]     #43 std::__1::function<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:2560:12 (libmaster.so+0xcf2419)
[m-1]     #44 yb::Thread::SuperviseThread(void*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:774:3 (libyb_util.so+0x6a3008)
[m-1] 
[m-1]   Previous read of size 1 at 0x7b0800109560 by thread T28 (mutexes: read M2822, read M2820):
[m-1]     #0 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) <null> (libc++.so.1+0x7d464)
[m-1]     #1 google::protobuf::internal::ArenaStringPtr::CreateInstanceNoArena(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/include/google/protobuf/arenastring.h:312:16 (libmaster.so+0x83d57a)
[m-1]     #2 google::protobuf::internal::ArenaStringPtr::SetNoArena(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/include/google/protobuf/arenastring.h:213:7 (libmaster.so+0x83d4a0)
[m-1]     #3 google::protobuf::internal::ArenaStringPtr::AssignWithDefault(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const*, google::protobuf::internal::ArenaStringPtr) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/include/google/protobuf/arenastring.h:330:5 (liblog_proto.so+0x1f9aa)
[m-1]     #4 yb::HostPortPB::MergeFrom(yb::HostPortPB const&) ${BUILD_ROOT}/src/yb/common/common_net.pb.cc:478:13 (libyb_common_proto.so+0x370409)
[m-1]     #5 google::protobuf::internal::GenericTypeHandler<yb::HostPortPB>::Merge(yb::HostPortPB const&, yb::HostPortPB*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/include/google/protobuf/repeated_field.h:652:7 (libmaster.so+0x99e3d0)
[m-1]     #6 void google::protobuf::internal::RepeatedPtrFieldBase::MergeFromInnerLoop<google::protobuf::RepeatedPtrField<yb::HostPortPB>::TypeHandler>(void**, void**, int, int) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/include/google/protobuf/repeated_field.h:1648:5 (libmaster.so+0x99e365)
[m-1]     #7 google::protobuf::internal::RepeatedPtrFieldBase::MergeFromInternal(google::protobuf::internal::RepeatedPtrFieldBase const&, void (google::protobuf::internal::RepeatedPtrFieldBase::*)(void**, void**, int, int)) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/include/google/protobuf/repeated_field.h:1619:3 (libmaster.so+0x8667f5)
[m-1]     #8 void google::protobuf::internal::RepeatedPtrFieldBase::MergeFrom<google::protobuf::RepeatedPtrField<yb::HostPortPB>::TypeHandler>(google::protobuf::internal::RepeatedPtrFieldBase const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/include/google/protobuf/repeated_field.h:1607:3 (libmaster.so+0x99e243)
[m-1]     #9 google::protobuf::RepeatedPtrField<yb::HostPortPB>::MergeFrom(google::protobuf::RepeatedPtrField<yb::HostPortPB> const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/include/google/protobuf/repeated_field.h:2108:25 (libmaster.so+0x99e1a0)
[m-1]     #10 yb::master::BlacklistPB::MergeFrom(yb::master::BlacklistPB const&) ${BUILD_ROOT}/src/yb/master/catalog_entity_info.pb.cc:7264:10 (libmaster_proto.so+0x90e77b)
[m-1]     #11 yb::master::BlacklistPB::CopyFrom(yb::master::BlacklistPB const&) ${BUILD_ROOT}/src/yb/master/catalog_entity_info.pb.cc:7288:3 (libmaster_proto.so+0x90e88d)
[m-1]     #12 yb::master::BlacklistPB::operator=(yb::master::BlacklistPB const&) ${BUILD_ROOT}/src/yb/master/catalog_entity_info.pb.h:2401:5 (libmaster.so+0xa7cdf0)
[m-1]     #13 yb::master::PerTableLoadState::SetLeaderBlacklist(yb::master::BlacklistPB const&) ${BUILD_ROOT}/../../src/yb/master/cluster_balance_util.h:283:23 (libmaster.so+0xa7a087)
[m-1]     #14 yb::master::ClusterLoadBalancer::InitializeTSDescriptors() ${BUILD_ROOT}/../../src/yb/master/cluster_balance.cc:1451:11 (libmaster.so+0xa6f8d0)
[m-1]     #15 yb::master::ClusterLoadBalancer::RunLoadBalancerWithOptions(yb::master::Options*) ${BUILD_ROOT}/../../src/yb/master/cluster_balance.cc:414:5 (libmaster.so+0xa6e1c6)
[m-1]     #16 yb::master::ClusterLoadBalancer::RunLoadBalancer(yb::master::Options*) ${BUILD_ROOT}/../../src/yb/master/cluster_balance.cc:598:3 (libmaster.so+0xa71553)
[m-1]     #17 yb::master::CatalogManagerBgTasks::Run() ${BUILD_ROOT}/../../src/yb/master/catalog_manager_bg_tasks.cc:182:51 (libmaster.so+0xa575c8)
[m-1]     #18 decltype(*(std::__1::forward<yb::master::CatalogManagerBgTasks*&>(fp0)).*fp()) std::__1::__invoke<void (yb::master::CatalogManagerBgTasks::*&)(), yb::master::CatalogManagerBgTasks*&, void>(void (yb::master::CatalogManagerBgTasks::*&)(), yb::master::CatalogManagerBgTasks*&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/type_traits:3635:1 (libmaster.so+0xa5b5dc)
[m-1]     #19 std::__1::__bind_return<void (yb::master::CatalogManagerBgTasks::*)(), std::__1::tuple<yb::master::CatalogManagerBgTasks*>, std::__1::tuple<>, __is_valid_bind_return<void (yb::master::CatalogManagerBgTasks::*)(), std::__1::tuple<yb::master::CatalogManagerBgTasks*>, std::__1::tuple<> >::value>::type std::__1::__apply_functor<void (yb::master::CatalogManagerBgTasks::*)(), std::__1::tuple<yb::master::CatalogManagerBgTasks*>, 0ul, std::__1::tuple<> >(void (yb::master::CatalogManagerBgTasks::*&)(), std::__1::tuple<yb::master::CatalogManagerBgTasks*>&, std::__1::__tuple_indices<0ul>, std::__1::tuple<>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:2857:12 (libmaster.so+0xa5b549)
[m-1]     #20 std::__1::__bind_return<void (yb::master::CatalogManagerBgTasks::*)(), std::__1::tuple<yb::master::CatalogManagerBgTasks*>, std::__1::tuple<>, __is_valid_bind_return<void (yb::master::CatalogManagerBgTasks::*)(), std::__1::tuple<yb::master::CatalogManagerBgTasks*>, std::__1::tuple<> >::value>::type std::__1::__bind<void (yb::master::CatalogManagerBgTasks::* const&)(), yb::master::CatalogManagerBgTasks* const&>::operator()<>() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:2890:20 (libmaster.so+0xa5b4f1)
[m-1]     #21 decltype(std::__1::forward<std::__1::__bind<void (yb::master::CatalogManagerBgTasks::* const&)(), yb::master::CatalogManagerBgTasks* const&>&>(fp)()) std::__1::__invoke<std::__1::__bind<void (yb::master::CatalogManagerBgTasks::* const&)(), yb::master::CatalogManagerBgTasks* const&>&>(std::__1::__bind<void (yb::master::CatalogManagerBgTasks::* const&)(), yb::master::CatalogManagerBgTasks* const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/type_traits:3694:1 (libmaster.so+0xa5b481)
[m-1]     #22 void std::__1::__invoke_void_return_wrapper<void, true>::__call<std::__1::__bind<void (yb::master::CatalogManagerBgTasks::* const&)(), yb::master::CatalogManagerBgTasks* const&>&>(std::__1::__bind<void (yb::master::CatalogManagerBgTasks::* const&)(), yb::master::CatalogManagerBgTasks* const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/__functional_base:348:9 (libmaster.so+0xa5b411)
[m-1]     #23 std::__1::__function::__alloc_func<std::__1::__bind<void (yb::master::CatalogManagerBgTasks::* const&)(), yb::master::CatalogManagerBgTasks* const&>, std::__1::allocator<std::__1::__bind<void (yb::master::CatalogManagerBgTasks::* const&)(), yb::master::CatalogManagerBgTasks* const&> >, void ()>::operator()() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:1558:16 (libmaster.so+0xa5b3d1)
[m-1]     #24 std::__1::__function::__func<std::__1::__bind<void (yb::master::CatalogManagerBgTasks::* const&)(), yb::master::CatalogManagerBgTasks* const&>, std::__1::allocator<std::__1::__bind<void (yb::master::CatalogManagerBgTasks::* const&)(), yb::master::CatalogManagerBgTasks* const&> >, void ()>::operator()() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:1732:12 (libmaster.so+0xa5a05d)
[m-1]     #25 std::__1::__function::__value_func<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:1885:16 (libmaster.so+0xd09084)
[m-1]     #26 std::__1::function<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/functional:2560:12 (libmaster.so+0xcf2419)
[m-1]     #27 yb::Thread::SuperviseThread(void*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:774:3 (libyb_util.so+0x6a3008)

@bmatican bmatican added priority/high High Priority and removed priority/medium Medium priority issue labels Sep 6, 2022
sanketkedia added a commit that referenced this issue Sep 12, 2022
Summary:
Catalog Manager returns both data blacklist and leader blacklist by reference to the load balancer.
If we try to modify the cluster config at the same time then it creates a race
and might even lead to the old blacklist getting destroyed before the LB consumes it. Fixed by
making a deep copy of the blacklist under the CoW read lock.

This diff also gets rid of the mutex lock guarding the shared_ptr to the cluster config. We only
modify this pointer on a catalog load so we don't need any guard since both internal and external
accesses to the cluster config happen under the scoped leader shared lock and during catalog loading
we acquire an exclusive leader lock.

Test Plan: /yb_build.sh tsan --clang12 --cxx_test cassandra_cpp_driver-test --gtest-filter CppCassandraDriverTest.TestTableBackfillWithLeaderMoves

Reviewers: hsunder, slingam, nicolas, bogdan

Reviewed By: bogdan

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D19425
@rthallamko3
Copy link
Contributor

@sanketkedia , Do you plan to fix the failures in the test as part of this GH issue itself or should this be closed a new one filed? It seems like there are failures in the test even after your change, but the failure seems to be different from the tsan race.


09/17/2022 01:48 | alma8-master-clang12-asan/1318/ |   | Log: https://jenkins.dev.yugabyte.com/job/github-yugabyte-db-alma8-master-clang12-asan/1318/artifact/build/asan-clang12-dynamic-ninja/yb-test-logs/tests-integration-tests__cassandra_cpp_driver-test/CppCassandraDriverTest_TestTableBackfillWithLeaderMoves.log  ../../src/yb/integration-tests/cassandra_cpp_driver-test.cc:1259 Expected: (index_table_size) <= (kUpperBound), actual: 103 vs 100
-- | -- | -- | --
09/16/2022 22:03 | centos-master-clang14-tsan/6/ |   | Log: https://jenkins.dev.yugabyte.com/job/github-yugabyte-db-centos-master-clang14-tsan/6/artifact/build/tsan-clang14-dynamic-ninja/yb-test-logs/tests-integration-tests__cassandra_cpp_driver-test/CppCassandraDriverTest_TestTableBackfillWithLeaderMoves.log  ../../src/yb/integration-tests/cassandra_cpp_driver-test.cc:1259 Expected: (index_table_size) <= (kUpperBound), actual: 101 vs 100

09/17/2022 01:48	[alma8-master-clang12-asan/1318/](https://jenkins.dev.yugabyte.com/job/github-yugabyte-db-alma8-master-clang12-asan/1318/)		Log: https://jenkins.dev.yugabyte.com/job/github-yugabyte-db-alma8-master-clang12-asan/1318/artifact/build/asan-clang12-dynamic-ninja/yb-test-logs/tests-integration-tests__cassandra_cpp_driver-test/CppCassandraDriverTest_TestTableBackfillWithLeaderMoves.log

../../src/yb/integration-tests/cassandra_cpp_driver-test.cc:1259
Expected: (index_table_size) <= (kUpperBound), actual: 103 vs 100
09/16/2022 22:03	[centos-master-clang14-tsan/6/](https://jenkins.dev.yugabyte.com/job/github-yugabyte-db-centos-master-clang14-tsan/6/)		Log: https://jenkins.dev.yugabyte.com/job/github-yugabyte-db-centos-master-clang14-tsan/6/artifact/build/tsan-clang14-dynamic-ninja/yb-test-logs/tests-integration-tests__cassandra_cpp_driver-test/CppCassandraDriverTest_TestTableBackfillWithLeaderMoves.log

../../src/yb/integration-tests/cassandra_cpp_driver-test.cc:1259
Expected: (index_table_size) <= (kUpperBound), actual: 101 vs 100

sanketkedia added a commit that referenced this issue Nov 23, 2022
…in Cluster Config

Summary:
Original commit: 36a6501 / D19425
Catalog Manager returns both data blacklist and leader blacklist by reference to the load balancer.
If we try to modify the cluster config at the same time then it creates a race
and might even lead to the old blacklist getting destroyed before the LB consumes it. Fixed by
making a deep copy of the blacklist under the CoW read lock.

This diff also gets rid of the mutex lock guarding the shared_ptr to the cluster config. We only
modify this pointer on a catalog load so we don't need any guard since both internal and external
accesses to the cluster config happen under the scoped leader shared lock and during catalog loading
we acquire an exclusive leader lock.

Test Plan: /yb_build.sh tsan --clang12 --cxx_test cassandra_cpp_driver-test --gtest-filter CppCassandraDriverTest.TestTableBackfillWithLeaderMoves

Reviewers: hsunder, slingam, bogdan, nicolas

Reviewed By: nicolas

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D21232
@bmatican
Copy link
Contributor Author

bmatican commented Feb 1, 2023

Closing this as the TSAN race is fixed. Can file a new one, if still is still flaky, for any index specific issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/docdb YugabyteDB core features kind/bug This issue is a bug kind/failing-test Tests and testing infra priority/high High Priority
Projects
None yet
Development

No branches or pull requests

5 participants