[Enhancement] Bump FE/BE's hadoop to 3.4.0 #45312
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![starrocks-cr[bot]](https://avatars.githubusercontent.com/in/404334?s=60&v=4){kind=small}
| hiveResourceUri.setResourceType(ResourceType.valueOf(catalogResourceUri.getResourceType())); | ||
| hiveResourceUri.setUri(catalogResourceUri.uri()); | ||
| if (catalogResourceUri.resourceType() != null) { | ||
| hiveResourceUri.setResourceType(ResourceType.valueOf(catalogResourceUri.resourceType().toString())); | ||
| } | ||
| hiveResourceUriList.add(hiveResourceUri); | ||
| } |
| } | ||
| catalogResourceUriList.add(catalogResourceUri); | ||
| catalogResourceUriList.add(catalogResourceUri.build()); | ||
| } | ||
| return catalogResourceUriList; | ||
| } |
| @@ -1627,7 +1627,7 @@ public void renamePartitionInCatalog(String databaseName, String tableName, List | |||
| try { | |||
| PartitionInput partitionInput = GlueInputConverter.convertToPartitionInput(newPartition); | |||
| glueMetastore.updatePartition(databaseName, tableName, partitionValues, partitionInput); | |||
| } catch (AmazonServiceException e) { | |||
| } catch (SdkException e) { | |||
| throw CatalogToHiveConverter.wrapInHiveException(e); | |||
| } | |||
| } | |||
imay
reviewed
May 9, 2024
| <dependency> | ||
| <groupId>software.amazon.awssdk</groupId> | ||
| <artifactId>bundle</artifactId> | ||
| <version>${awsv2.version}</version> |
There was a problem hiding this comment.
exclude unused dependency
There was a problem hiding this comment.
hard do it, hadoop-aws will include bundle automatically.
aws SDK v2 is introduced by hadoop-aws : 3.4.0 Because we upgraded Hadoop, we also upgraded aws. |
Smith-Cruise
force-pushed
the
upgrade-s3-sdk
branch
3 times, most recently
from
bf6a82e to
06b5f1d
Compare
Smith-Cruise
changed the title
Smith-Cruise
force-pushed
the
upgrade-s3-sdk
branch
4 times, most recently
from
c63a45e to
a5e4fc2
Compare
Smith-Cruise
changed the title
Smith-Cruise
force-pushed
the
upgrade-s3-sdk
branch
from
9009ab0 to
2214e37
Compare
build.sh
Outdated
| @@ -519,6 +519,7 @@ if [ ${BUILD_BE} -eq 1 ]; then | |||
| cp -r -p ${STARROCKS_HOME}/java-extensions/iceberg-metadata-reader/target/iceberg-reader-lib ${STARROCKS_OUTPUT}/be/lib/ | |||
| cp -r -p ${STARROCKS_HOME}/java-extensions/iceberg-metadata-reader/target/starrocks-iceberg-metadata-reader.jar ${STARROCKS_OUTPUT}/be/lib/jni-packages | |||
| cp -r -p ${STARROCKS_HOME}/java-extensions/iceberg-metadata-reader/target/starrocks-iceberg-metadata-reader.jar ${STARROCKS_OUTPUT}/be/lib/iceberg-reader-lib | |||
| cp -r -p ${STARROCKS_HOME}/java-extensions/jni-runtime/target/jni-runtime-lib ${STARROCKS_OUTPUT}/be/lib/ | |||
There was a problem hiding this comment.
we have existed jni-package. I think it more appropriate to name it common-runtime-lib
java-extensions/jni-runtime/pom.xml
Outdated
| <version>1.0-SNAPSHOT</version> | ||
| </parent> | ||
|
|
||
| <artifactId>jni-runtime</artifactId> |
| // The UrlConnectionHttpClient is introduced by #16602, and it causes the exception. | ||
| // So we set the default HTTP client to UrlConnectionHttpClient. | ||
| // TODO: remove this after we remove ApacheHttpClient | ||
| private static void setAWSHttpClient() { |
There was a problem hiding this comment.
remove it in the iceberg_metadata_reader
Smith-Cruise
force-pushed
the
upgrade-s3-sdk
branch
from
2214e37 to
a46fbf3
Compare
Signed-off-by: Smith Cruise <chendingchao1@126.com>
Smith-Cruise
force-pushed
the
upgrade-s3-sdk
branch
from
ed65cc9 to
0c8c2f0
Compare
|
[FE Incremental Coverage Report]❌ fail : 112 / 496 (22.58%) file detail
|
[BE Incremental Coverage Report]✅ pass : 0 / 0 (0%) |
stephen-shelby
approved these changes
May 16, 2024
Youngwb
approved these changes
May 16, 2024
|
@mergify backport branch-3.3 |
✅ Backports have been created
|
24 tasks
mergify bot
pushed a commit
that referenced
this pull request
May 16, 2024
Why I'm doing: For the CVE problem, we need to upgrade Hadoop SDK from 3.3.6 -> 3.4.0 It will introduce aws java SDK v2, so we can delete SDK v1. Signed-off-by: Smith Cruise <chendingchao1@126.com> (cherry picked from commit 0fcf3d7)
42 tasks
wanpengfei-git
pushed a commit
that referenced
this pull request
May 16, 2024
42 tasks
node
pushed a commit
to vivo/starrocks
that referenced
this pull request
May 17, 2024
Why I'm doing: For the CVE problem, we need to upgrade Hadoop SDK from 3.3.6 -> 3.4.0 It will introduce aws java SDK v2, so we can delete SDK v1. Signed-off-by: Smith Cruise <chendingchao1@126.com>
24 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why I'm doing:
For the CVE problem, we need to upgrade Hadoop SDK from 3.3.6 -> 3.4.0
It will introduce aws java SDK v2, so we can delete SDK v1.
What I'm doing:
Do it.
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: