Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bugfix](thirdparty) Upgrade aws s3 sdk to prevent mem leak #25106

Merged
merged 2 commits into from
Oct 9, 2023
Merged

[bugfix](thirdparty) Upgrade aws s3 sdk to prevent mem leak #25106

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7c6b8cb
upgrade aws s3 sdk
ByteYue Oct 8, 2023
aed325f
add patch
ByteYue Oct 8, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions be/cmake/thirdparty.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,7 @@ add_thirdparty(aws-c-http LIB64)
add_thirdparty(aws-c-mqtt LIB64)
add_thirdparty(aws-checksums LIB64)
add_thirdparty(aws-c-s3 LIB64)
add_thirdparty(aws-c-sdkutils LIB64)
if (NOT OS_MACOSX)
add_thirdparty(aws-s2n LIBNAME "lib/libs2n.a")
endif()
Expand Down
3 changes: 3 additions & 0 deletions thirdparty/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@

This file contains version of the third-party dependency libraries in the build-env image. The docker build-env image is apache/doris, and the tag is `build-env-${version}`

## v20231008
- Modified: aws sdk 1.9.211 -> 1.11.119

## v20230824
- Modified: protobuf 3.15.0 -> 21.11
- Modified: glog 0.4.0 -> 0.6.0
Expand Down
7 changes: 4 additions & 3 deletions thirdparty/download-thirdparty.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,12 +386,13 @@ echo "Finished patching ${HYPERSCAN_SOURCE}"

cd "${TP_SOURCE_DIR}/${AWS_SDK_SOURCE}"
if [[ ! -f "${PATCHED_MARK}" ]]; then
if [[ "${AWS_SDK_SOURCE}" == "aws-sdk-cpp-1.9.272" ]]; then
if wget --no-check-certificate -q https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/aws-crt-cpp-1.9.272.tar.gz -O aws-crt-cpp-1.9.272.tar.gz; then
tar xzf aws-crt-cpp-1.9.272.tar.gz
if [[ "${AWS_SDK_SOURCE}" == "aws-sdk-cpp-1.11.119" ]]; then
if wget --no-check-certificate -q https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/aws-crt-cpp-1.11.119.tar.gz -O aws-crt-cpp-1.11.119.tar.gz; then
tar xzf aws-crt-cpp-1.11.119.tar.gz
else
bash ./prefetch_crt_dependency.sh
fi
patch -p1 <"${TP_PATCH_DIR}/aws-sdk-cpp-1.11.119.patch"
else
bash ./prefetch_crt_dependency.sh
fi
Expand Down
52 changes: 52 additions & 0 deletions thirdparty/patches/aws-sdk-cpp-1.11.119.patch
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
diff --git a/generated/src/aws-cpp-sdk-s3-crt/source/model/CopyObjectRequest.cpp b/generated/src/aws-cpp-sdk-s3-crt/source/model/CopyObjectRequest.cpp
index 2aa7869b51..9ba8877ed8 100644
--- a/generated/src/aws-cpp-sdk-s3-crt/source/model/CopyObjectRequest.cpp
+++ b/generated/src/aws-cpp-sdk-s3-crt/source/model/CopyObjectRequest.cpp
@@ -173,7 +173,7 @@ Aws::Http::HeaderValueCollection CopyObjectRequest::GetRequestSpecificHeaders()
if(m_copySourceHasBeenSet)
{
ss << m_copySource;
- headers.emplace("x-amz-copy-source", URI::URLEncodePath(ss.str()));
+ headers.emplace("x-amz-copy-source", ss.str());
ss.str("");
}

diff --git a/generated/src/aws-cpp-sdk-s3-crt/source/model/UploadPartCopyRequest.cpp b/generated/src/aws-cpp-sdk-s3-crt/source/model/UploadPartCopyRequest.cpp
index f301c47606..7f16e6afcd 100644
--- a/generated/src/aws-cpp-sdk-s3-crt/source/model/UploadPartCopyRequest.cpp
+++ b/generated/src/aws-cpp-sdk-s3-crt/source/model/UploadPartCopyRequest.cpp
@@ -112,7 +112,7 @@ Aws::Http::HeaderValueCollection UploadPartCopyRequest::GetRequestSpecificHeader
if(m_copySourceHasBeenSet)
{
ss << m_copySource;
- headers.emplace("x-amz-copy-source", URI::URLEncodePath(ss.str()));
+ headers.emplace("x-amz-copy-source", ss.str());
ss.str("");
}

diff --git a/generated/src/aws-cpp-sdk-s3/source/model/CopyObjectRequest.cpp b/generated/src/aws-cpp-sdk-s3/source/model/CopyObjectRequest.cpp
index e1862b22ff..6bd6d0aea9 100644
--- a/generated/src/aws-cpp-sdk-s3/source/model/CopyObjectRequest.cpp
+++ b/generated/src/aws-cpp-sdk-s3/source/model/CopyObjectRequest.cpp
@@ -173,7 +173,7 @@ Aws::Http::HeaderValueCollection CopyObjectRequest::GetRequestSpecificHeaders()
if(m_copySourceHasBeenSet)
{
ss << m_copySource;
- headers.emplace("x-amz-copy-source", URI::URLEncodePath(ss.str()));
+ headers.emplace("x-amz-copy-source", ss.str());
ss.str("");
}

diff --git a/generated/src/aws-cpp-sdk-s3/source/model/UploadPartCopyRequest.cpp b/generated/src/aws-cpp-sdk-s3/source/model/UploadPartCopyRequest.cpp
index f918909a86..16af4d2797 100644
--- a/generated/src/aws-cpp-sdk-s3/source/model/UploadPartCopyRequest.cpp
+++ b/generated/src/aws-cpp-sdk-s3/source/model/UploadPartCopyRequest.cpp
@@ -112,7 +112,7 @@ Aws::Http::HeaderValueCollection UploadPartCopyRequest::GetRequestSpecificHeader
if(m_copySourceHasBeenSet)
{
ss << m_copySource;
- headers.emplace("x-amz-copy-source", URI::URLEncodePath(ss.str()));
+ headers.emplace("x-amz-copy-source", ss.str());
ss.str("");
}

8 changes: 4 additions & 4 deletions thirdparty/vars.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -346,10 +346,10 @@ BOOTSTRAP_TABLE_CSS_FILE="bootstrap-table.min.css"
BOOTSTRAP_TABLE_CSS_MD5SUM="23389d4456da412e36bae30c469a766a"

# aws sdk
AWS_SDK_DOWNLOAD="https://github.com/aws/aws-sdk-cpp/archive/refs/tags/1.9.272.tar.gz"
AWS_SDK_NAME="aws-sdk-cpp-1.9.272.tar.gz"
AWS_SDK_SOURCE="aws-sdk-cpp-1.9.272"
AWS_SDK_MD5SUM="3a9190e3d14b81bfdd9c41a89a419202"
AWS_SDK_DOWNLOAD="https://github.com/aws/aws-sdk-cpp/archive/refs/tags/1.11.119.tar.gz"
AWS_SDK_NAME="aws-sdk-cpp-1.11.119.tar.gz"
AWS_SDK_SOURCE="aws-sdk-cpp-1.11.119"
AWS_SDK_MD5SUM="3cd8bd51d39dc207a243a2074d11f439"

# tsan_header
TSAN_HEADER_DOWNLOAD="https://gcc.gnu.org/git/?p=gcc.git;a=blob_plain;f=libsanitizer/include/sanitizer/tsan_interface_atomic.h;hb=refs/heads/releases/gcc-7"
Expand Down