Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BugFix] Fix the thrift library conflict in some cases by linking the required cachelib dependencies statically #16674

Merged
merged 2 commits into from
Jan 17, 2023

Conversation

GavinMar
Copy link
Contributor

@GavinMar GavinMar commented Jan 16, 2023

What type of PR is this:

  • BugFix
  • Feature
  • Enhancement
  • Refactor
  • UT
  • Doc
  • Tool

Which issues of this PR fixes :

Fixes #

Problem Summary(Required) :

When linking the cachelib dynamically by find_package, all related libraries will be linked. While some of them are not required in our usage scenarios, and even lead to dependency conflicts in some environments.

So, we choose to only link the required cachelib dependencies statically to avoid unnecessary conflicts. Also, with this change, it is no need to package lots of dynamic libraries in our be output.

Checklist:

  • I have added test cases for my bug fix or my new feature
  • This pr will affect users' behaviors
  • This pr needs user documentation (for new or modified features or behaviors)
    • I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

  • I have checked the version labels which the pr will be auto backported to target branch
    • 2.5
    • 2.4
    • 2.3
    • 2.2

Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

#ifndef JEMALLOC_NO_RENAME
#define JEMALLOC_NO_RENAME 1
#endif

#include <cachelib/allocator/CacheAllocator.h>

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'cachelib/allocator/CacheAllocator.h' file not found [clang-diagnostic-error]

#include <cachelib/allocator/CacheAllocator.h>
         ^

@GavinMar
Copy link
Contributor Author

run starrocks_admit_test image

@GavinMar
Copy link
Contributor Author

run starrocks_admit_test image

@GavinMar
Copy link
Contributor Author

run starrocks_be_unittest

@GavinMar
Copy link
Contributor Author

run starrocks_admit_test

@GavinMar
Copy link
Contributor Author

run starrocks_fe_unittest

@dirtysalt
Copy link
Contributor

run starrocks_fe_unittest

@wanpengfei-git wanpengfei-git added the Approved Ready to merge label Jan 17, 2023
@wanpengfei-git
Copy link
Collaborator

run starrocks_admit_test

@wanpengfei-git
Copy link
Collaborator

[FE PR Coverage Check]

😍 pass : 0 / 0 (0%)

@imay imay merged commit 3be64e4 into StarRocks:main Jan 17, 2023
@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.5% 0.5% Duplication

@bigsusu123
Copy link

bigsusu123 commented Mar 15, 2023

hi @GavinMar

I'm trying to customize a cachelib for our own purpose, but not knowing how to build a static cachelib so that it can still integrate into SR.

I'm using this version of cachelib: https://github.com/facebook/CacheLib/releases/tag/v2023.01.30.00,
and adding -DBUILD_SHARED_LIBS=OFF in contrib/build-package.sh to change default dynamic lib to static lib.
But I find the outcome .a files are significantly smaller than in https://cdn-thirdparty.starrocks.com/cachelib/cachelib-20230130.tar.gz:

my libs:

libcachelib_allocator.a 48M
libcachelib_common.a 4.4M

cdn-thirdparty:

libcachelib_allocator.a 170M
libcachelib_common.a 17M

am I doing it correctly ? can you please share some notes on how to build the cachelib?

thanks!

@GavinMar
Copy link
Contributor Author

hi @GavinMar

I'm trying to customize a cachelib for our own purpose, but not knowing how to build a static cachelib so that it can still integrate into SR.

I'm using this version of cachelib: https://github.com/facebook/CacheLib/releases/tag/v2023.01.30.00, and adding -DBUILD_SHARED_LIBS=OFF in contrib/build-package.sh to change default dynamic lib to static lib. But I find the outcome .a files are significantly smaller than in https://cdn-thirdparty.starrocks.com/cachelib/cachelib-20230130.tar.gz:

my libs:

libcachelib_allocator.a 48M libcachelib_common.a 4.4M

cdn-thirdparty:

libcachelib_allocator.a 170M libcachelib_common.a 17M

am I doing it correctly ? can you please share some notes on how to build the cachelib?

thanks!

Yes, I also built cachelib by adding DBUILD_SHARED_LIBS=OFF in contrib/build-package.sh. I'm not quite sure why your output is so much smaller than mine because you choose a newer cachelib version. So, if you could build all libraries successfully, I think you can try to integrate it into SR and check whether the block cache can work as expected.

@bigsusu123
Copy link

hi @GavinMar

thanks for the reply.
may i know which cachelib version did you use?

@GavinMar
Copy link
Contributor Author

hi @GavinMar

thanks for the reply. may i know which cachelib version did you use?

I used the version v2023.01.23 and I think your version is also ok. And you'd better build the cachelib with the same toolkits, such as g++ version, to avoid some abi compatibility problems.

@bigsusu123
Copy link

thank you so much for the tips! I'll give another shot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants