HADOOP-19399. S3A: support CRT client. #7443
Open
+774
−232
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
💔 -1 overall
This message was automatically generated. |
ahmarsuhail
commented
Mar 3, 2025
| @@ -0,0 +1,70 @@ | |||
| package org.apache.hadoop.fs.s3a.impl; | |||
|
|
|||
There was a problem hiding this comment.
@steveloughran starting to work on the CRT changes, wanted to get your opinion on this. What do you think about doing some refactoring around this region logic, so we have our own builder and container class for AWSRegionEndpointInformation which all clients can get information out of.
Without this you have to duplicate the endpoint region logic for CRT and the other clients since they use different builder classes. CRT uses S3CrtAsyncClientBuilder which does not extend S3BaseClientBuilder and so you can't use the existing method..
There was a problem hiding this comment.
Pulling out is good.
we need to fix that region resolution logic
- work in ec2/k8s without going out of region -automatically
- add an option to use the default chain without the "add a space" trick, e.g a region called "(sdk)". Not backwards compatible, but makes clear what happens.
|
💔 -1 overall
This message was automatically generated. |
ahmarsuhail
force-pushed
the
HADOOP-19399-crt-client-support
branch
from
badf0db to
032c9a8
Compare
ahmarsuhail
commented
Mar 7, 2025
| @@ -324,6 +326,19 @@ | |||
| </properties> | |||
| </profile> | |||
|
|
|||
| <!-- Use the S3 CRT client --> | |||
There was a problem hiding this comment.
this doesn't seem to be working btw, ran mvn clean verify -Dit.test=ITestAwsSdkWorkarounds -Dtest=none -Pcrt as I expect ITestAwsSdkWorkarounds to fail when using CRT, but it still passes. Need to look
There was a problem hiding this comment.
ITestAwsSdkWorkarounds stops looking for the error text with the sdk update patch
ahmarsuhail
commented
Mar 7, 2025
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java
Show resolved
Hide resolved
|
@steveloughran @mukund-thakur @shameersss1 Could you please review? Internal TPC-DS benchmarks for AAL seem to be consistently showing better numbers with CRT, possibly due to the CRT's enhanced connection management, so want to get this in for 3.4.2. |
ahmarsuhail
commented
Mar 7, 2025
| durationTrackerFactory, | ||
| STORE_CLIENT_CREATION.getSymbol(), | ||
| () -> clientFactory.createS3AsyncClient(getUri(), clientCreationParameters)); | ||
| return trackDurationOfOperation(durationTrackerFactory, STORE_CLIENT_CREATION.getSymbol(), |
There was a problem hiding this comment.
this should actually go into the client factory, not here. moving this over
rajdchak
reviewed
Mar 7, 2025
| if (regionEndpointInformation.getEndpoint() == null) { | ||
| s3CrtAsyncClientBuilder.endpointOverride(regionEndpointInformation.getEndpoint()); | ||
| } | ||
|
|
There was a problem hiding this comment.
This doesn't look right. We are doing a null check for both the region and the endpoint and then setting to that if null?
There was a problem hiding this comment.
thanks, it was not right :) fixed!
There was a problem hiding this comment.
i'm thinking we need a special region "sdk" to let the sdk take over region resolution; would that help here too?
rajdchak
reviewed
Mar 7, 2025
| @@ -0,0 +1,232 @@ | |||
| /* | |||
There was a problem hiding this comment.
Can we not modify the configureEndpointAndRegion method in DefaultS3ClientFactory class to support AWSRegionEndpointInformation and not create a new class for this?
There was a problem hiding this comment.
prefer to move it out to a separate class, this logic doesn't really belong in the client factory imo
There was a problem hiding this comment.
this stuff is getting to complicated and the endpoint/region code a significant source of pain. Isolation, more tests etc would be good.
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Great to hear we see improvements in the performance. Just curious, Was it for Parquet file format ? And % improvement ? |
shameersss1
reviewed
Mar 10, 2025
| } | ||
|
|
||
| if (regionEndpointInformation.getEndpoint() != null) { | ||
| s3CrtAsyncClientBuilder.endpointOverride(regionEndpointInformation.getEndpoint()); |
There was a problem hiding this comment.
Is setting both endpoint and region expected ? Won't the client throw exception in that case ?
There was a problem hiding this comment.
yeah, I haven't made any changes to the logic, just refactored it a bit. We've always had the ability for uses to set both the endpoint and the region.. I think maybe useful if you want to us a specific endpoint for a region (eg: fips enabled?) .. this whole logic does need a bit of a rethink and re-write, see: https://issues.apache.org/jira/browse/HADOOP-19470
shameersss1
reviewed
Mar 10, 2025
| @@ -356,7 +362,7 @@ public PutObjectRequest.Builder newPutObjectRequestBuilder(String key, | |||
| } | |||
|
|
|||
| // Set the timeout for object uploads but not directory markers. | |||
There was a problem hiding this comment.
edit the comment as well
|
@ahmarsuhail : I see an effort to upgrade SDK V2 : https://issues.apache.org/jira/browse/HADOOP-19485 I am not sure will that cause any conflicts here. Wouldn't it be nice to start this after upgrade ? Is there any depedency between SDK version CRT version ? |
|
thanks @shameersss1, yeah agreed makes sense to wait for the SDK upgrade to go in first. i just wanted to get the code up for review, but will test again after the upgrade |
steveloughran
requested changes
Mar 11, 2025
| @@ -1767,6 +1767,48 @@ To disable checksum verification in `distcp`, use the `-skipcrccheck` option: | |||
| hadoop distcp -update -skipcrccheck -numListstatusThreads 40 /user/alice/datasets s3a://alice-backup/datasets | |||
| ``` | |||
|
|
|||
| ### <a name="distcp"></a> Using the S3 CRT client | |||
There was a problem hiding this comment.
pull out all crtclient stuff into its own doc
| <dependency> | ||
| <groupId>software.amazon.awssdk.crt</groupId> | ||
| <artifactId>aws-crt</artifactId> | ||
| <scope>compile</scope> |
There was a problem hiding this comment.
provided, unless it really is to be shipped
There was a problem hiding this comment.
lets leave at provided, but docs to indicate this can be left out if the option is disabled
There was a problem hiding this comment.
do you mean leave at compile ? or should I move to provided?
Also just to confirm, only difference is if it's provided it won't get included in the final release tar, and anyone who wants to use it has to put the jar there?
| public S3AsyncClient createS3AsyncClient(final URI uri, | ||
| final S3ClientCreationParameters parameters) throws IOException { | ||
| if (parameters.isCrtEnabled()) { | ||
| LOG_S3_CRT_ENABLED.info("The S3 CRT client is enabled"); |
There was a problem hiding this comment.
why not at debug? if everything works. no need to print anything
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
Show resolved
Hide resolved
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/RequestFactoryImpl.java
Show resolved
Hide resolved
|
|
||
| The [AWS CRT-based S3 client](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/crt-based-s3-client.html) | ||
| is built on top of the AWS Common Runtime (CRT), is an alternative S3 asynchronous client. It has | ||
| enhanced connection pool management, and can provide higher transfer from and to S3 due to its |
There was a problem hiding this comment.
higher transfer what? bandwidth, latency, support calls?.
also, s3a block output stream splits PUT requests, and vector io/analytics does the GET stuff, so what else does it offer? load balancing?
| @@ -0,0 +1,70 @@ | |||
| package org.apache.hadoop.fs.s3a.impl; | |||
|
|
|||
There was a problem hiding this comment.
Pulling out is good.
we need to fix that region resolution logic
- work in ec2/k8s without going out of region -automatically
- add an option to use the default chain without the "add a space" trick, e.g a region called "(sdk)". Not backwards compatible, but makes clear what happens.
| @@ -0,0 +1,232 @@ | |||
| /* | |||
There was a problem hiding this comment.
this stuff is getting to complicated and the endpoint/region code a significant source of pain. Isolation, more tests etc would be good.
| // region configuration was set to empty string. | ||
| // allow this if people really want it; it is OK to rely on this | ||
| // when deployed in EC2. | ||
| WARN_OF_DEFAULT_REGION_CHAIN.warn(SDK_REGION_CHAIN_IN_USE); |
There was a problem hiding this comment.
lets just log at debug
steveloughran
changed the title
steveloughran
requested changes
Mar 28, 2025
There was a problem hiding this comment.
commented
@ahmarsuhail I think you'd missed I'd already commented on it, so now I add more homework for you
hadoop-project/pom.xml
Outdated
| @@ -205,6 +205,7 @@ | |||
| <surefire.fork.timeout>900</surefire.fork.timeout> | |||
| <aws-java-sdk.version>1.12.720</aws-java-sdk.version> | |||
| <aws-java-sdk-v2.version>2.25.53</aws-java-sdk-v2.version> | |||
| <software.amazon.awssdk.crt.version>0.29.11</software.amazon.awssdk.crt.version> | |||
There was a problem hiding this comment.
change to aws-crt-client.version
| <dependency> | ||
| <groupId>software.amazon.awssdk.crt</groupId> | ||
| <artifactId>aws-crt</artifactId> | ||
| <scope>compile</scope> |
There was a problem hiding this comment.
lets leave at provided, but docs to indicate this can be left out if the option is disabled
| @@ -175,6 +163,39 @@ public S3AsyncClient createS3AsyncClient( | |||
| return s3AsyncClientBuilder.build(); | |||
| } | |||
|
|
|||
| private S3AsyncClient createS3CrtAsyncClient(URI uri, S3ClientCreationParameters parameters) | |||
There was a problem hiding this comment.
can we pull all async client creation into its own class, so as to be confident that there's no crt client references elsewhere in the code? I don't what it to become mandatory on the classpath
There was a problem hiding this comment.
These CRT classes referenced here are all included in the SDK bundle. It's just that if you try to initialise it, and you don't have the CRT dependency on the classpath it'll fail. do you think we still should need to move?
There was a problem hiding this comment.
Safe if these are the only places of this source file where the crt classes are referenced.
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
Show resolved
Hide resolved
| @@ -81,6 +81,7 @@ S3Client createS3Client(URI uri, | |||
| S3AsyncClient createS3AsyncClient(URI uri, | |||
| S3ClientCreationParameters parameters) throws IOException; | |||
|
|
|||
|
|
|||
| region = Region.of(AWS_S3_DEFAULT_REGION); | ||
| builder.withRegion(region); | ||
| origin = "cross region access fallback"; | ||
| } else if (configuredRegion.isEmpty()) { |
There was a problem hiding this comment.
proposed, extra regiond
- "sdk" which explicitly switches to sdk resolution
- "auto" which is "us doing the right thing"; document this may change across releases
| } | ||
|
|
||
|
|
||
| protected static URI getS3Endpoint(String endpoint, final Configuration conf) { |
There was a problem hiding this comment.
javadoc; highlight failure condition.
| } | ||
|
|
||
| try { | ||
| return new URI(endpoint); |
There was a problem hiding this comment.
need to think of unit tests to break this, e.g https://something/http://something-else
| @@ -59,6 +60,7 @@ public class ClientManagerImpl | |||
|
|
|||
| public static final Logger LOG = LoggerFactory.getLogger(ClientManagerImpl.class); | |||
|
|
|||
|
|
|||
|
This PR is dependent on the SDK upgrade: #7479 |
ahmarsuhail
commented
Mar 31, 2025
| @@ -165,14 +153,41 @@ public S3AsyncClient createS3AsyncClient( | |||
| configureClientBuilder(S3AsyncClient.builder(), parameters, conf, bucket) | |||
| .httpClientBuilder(httpClientBuilder); | |||
|
|
|||
| // multipart upload pending with HADOOP-19326. | |||
There was a problem hiding this comment.
safe to cut this now as SDK no longer complains with Multipart download is not yet supported. Instead use the CRT based S3 client for multipart download. when trying to do a Ranged GET with the S3 Async client and with >2.29.52
ahmarsuhail
force-pushed
the
HADOOP-19399-crt-client-support
branch
2 times, most recently
from
57ad5f2 to
e3122d0
Compare
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
@ahmarsuhail givnen the SDK upgrade is still awaiting approval, maybe you'd want to look at it |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
steveloughran
requested changes
Apr 30, 2025
| ## Enabling the CRT Client | ||
| The CRT client can be enabled as follows: | ||
|
|
||
| ``` |
| can be found | ||
| [here](https://aws.amazon.com/blogs/developer/introducing-crt-based-s3-client-and-the-s3-transfer-manager-in-the-aws-sdk-for-java-2-x/). | ||
|
|
||
| When making multiple parallel GET requests, using the CRT ensures load is evenly distributed across S3. This can be |
There was a problem hiding this comment.
S3 front end servers, S3 back end stores -or both?
| [here](https://aws.amazon.com/blogs/developer/introducing-crt-based-s3-client-and-the-s3-transfer-manager-in-the-aws-sdk-for-java-2-x/). | ||
|
|
||
| When making multiple parallel GET requests, using the CRT ensures load is evenly distributed across S3. This can be | ||
| useful for all three input streams available with versions > 3.4.2, as: |
| } | ||
|
|
||
| /** | ||
| * Builds user agent string |
| @@ -212,6 +246,27 @@ public static RetryPolicy.Builder createRetryPolicyBuilder(Configuration conf) { | |||
| return retryPolicyBuilder; | |||
| } | |||
|
|
|||
|
|
|||
| private static S3CrtProxyConfiguration mapCRTProxyConfiguration( | |||
| software.amazon.awssdk.http.nio.netty.ProxyConfiguration proxyConfiguration) { | |||
There was a problem hiding this comment.
this the same for the unshaded SDK JARs?
| builder.putHeader(REQUESTER_PAYS_HEADER, REQUESTER_PAYS_HEADER_VALUE); | ||
| } | ||
|
|
||
| builder.putHeader(USER_AGENT, userAgent); |
There was a problem hiding this comment.
not for this PR, but maybe in future we could add the auditor ? strings here so it gets through the CRT and all the way to cloudtrail
There was a problem hiding this comment.
yup, tracked in https://issues.apache.org/jira/browse/HADOOP-19365, will add audit support for CRT
| /** | ||
| * Requester pays header value. Value {@value}. | ||
| */ | ||
| public static final String REQUESTER_PAYS_HEADER_VALUE = "requester"; |
There was a problem hiding this comment.
make an InternalConstant unless there's a need to expose it
| @@ -175,6 +163,39 @@ public S3AsyncClient createS3AsyncClient( | |||
| return s3AsyncClientBuilder.build(); | |||
| } | |||
|
|
|||
| private S3AsyncClient createS3CrtAsyncClient(URI uri, S3ClientCreationParameters parameters) | |||
There was a problem hiding this comment.
Safe if these are the only places of this source file where the crt classes are referenced.
| @@ -50,6 +50,7 @@ full details. | |||
| * [Auditing Architecture](./auditing_architecture.html). | |||
| * [Testing](./testing.html) | |||
| * [S3Guard](./s3guard.html) | |||
| * [Using the S3 CRT Client](./crt_client.html). | |||
There was a problem hiding this comment.
- point to this in the reading.md file
- in performance.md, say that the analytics stream and crt client may also provide performance improvements
| /** | ||
| * Flag to enable the CRT client. Value {@value}. | ||
| */ | ||
| public static final String CRT_CLIENT_ENABLED = "fs.s3a.crt.enabled"; |
There was a problem hiding this comment.
can you export this in the filesystem hasPathCapability, and add to InternalContants.S3A_DYNAMIC_CAPABILITIES , for bucket-info and storediag. thanks.
steveloughran
requested changes
Apr 30, 2025
There was a problem hiding this comment.
tested s3 standard london, stream=analytics, crt=enabled. saw various failures
ot
This hasn't yet been rebased to the new SDK has it? some of the errors (e.g ITestAwsSdkWorkarounds) are already fixed so don't matter
- Test failures of vectored reads and elsewhere on
S3Client.makeMetaRequest has invalid options; Operation name must be set for MetaRequestType.DEFAULT.
ITestS3AAnalyticsAcceleratorStreamReading.testMalformedParquetFooter failed with "Caused by: java.lang.UnsupportedOperationException: Multipart download is not yet supported. Instead use the CRT based S3 client for multipart download."
other than these failures and a few minor comments, I'm happy
[ERROR] ITestS3AContractAnalyticsStreamVectoredRead.testConsecutiveRanges Time elapsed: 0.377 s <<< ERROR!
java.io.IOException: Client error accessing s3://stevel-london/test/vectored_file.txt
at software.amazon.s3.analyticsaccelerator.exceptions.ExceptionHandler.createIOException(ExceptionHandler.java:94)
at software.amazon.s3.analyticsaccelerator.exceptions.ExceptionHandler.lambda$static$2(ExceptionHandler.java:40)
at software.amazon.s3.analyticsaccelerator.exceptions.ExceptionHandler.lambda$toIOException$6(ExceptionHandler.java:74)
at java.util.Optional.map(Optional.java:215)
at software.amazon.s3.analyticsaccelerator.exceptions.ExceptionHandler.toIOException(ExceptionHandler.java:74)
at software.amazon.s3.analyticsaccelerator.S3SdkObjectClient.lambda$handleException$5(S3SdkObjectClient.java:197)
at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
at java.util.concurrent.CompletableFuture.uniExceptionallyStage(CompletableFuture.java:898)
at java.util.concurrent.CompletableFuture.exceptionally(CompletableFuture.java:2209)
at software.amazon.s3.analyticsaccelerator.S3SdkObjectClient.headObject(S3SdkObjectClient.java:139)
at software.amazon.s3.analyticsaccelerator.io.physical.data.MetadataStore.lambda$asyncGet$2(MetadataStore.java:123)
at java.util.HashMap.computeIfAbsent(HashMap.java:1128)
at java.util.Collections$SynchronizedMap.computeIfAbsent(Collections.java:2674)
at software.amazon.s3.analyticsaccelerator.io.physical.data.MetadataStore.asyncGet(MetadataStore.java:114)
at software.amazon.s3.analyticsaccelerator.io.physical.data.MetadataStore.get(MetadataStore.java:92)
at software.amazon.s3.analyticsaccelerator.io.physical.impl.PhysicalIOImpl.<init>(PhysicalIOImpl.java:87)
at software.amazon.s3.analyticsaccelerator.S3SeekableInputStreamFactory.createLogicalIO(S3SeekableInputStreamFactory.java:144)
at software.amazon.s3.analyticsaccelerator.S3SeekableInputStreamFactory.createStream(S3SeekableInputStreamFactory.java:113)
at org.apache.hadoop.fs.s3a.impl.streams.AnalyticsStream.<init>(AnalyticsStream.java:58)
at org.apache.hadoop.fs.s3a.impl.streams.AnalyticsStreamFactory.readObject(AnalyticsStreamFactory.java:70)
at org.apache.hadoop.fs.s3a.impl.S3AStoreImpl.readObject(S3AStoreImpl.java:964)
at org.apache.hadoop.fs.s3a.S3AFileSystem.executeOpen(S3AFileSystem.java:1922)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$null$42(S3AFileSystem.java:5503)
at org.apache.hadoop.util.LambdaUtils.eval(LambdaUtils.java:52)
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$openFileWithOptions$43(S3AFileSystem.java:5502)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: S3Client.makeMetaRequest has invalid options; Operation name must be set for MetaRequestType.DEFAULT.
at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111)
at software.amazon.awssdk.core.exception.SdkClientException.create(SdkClientException.java:47)
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:223)
at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.setLastException(RetryableStageHelper.java:218)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeRetryExecute(AsyncRetryableStage.java:182)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.lambda$attemptExecute$1(AsyncRetryableStage.java:159)
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
at java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:792)
at java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2153)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.attemptExecute(AsyncRetryableStage.java:156)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.maybeAttemptExecute(AsyncRetryableStage.java:136)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.execute(AsyncRetryableStage.java:95)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage.execute(AsyncRetryableStage.java:79)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage.execute(AsyncRetryableStage.java:44)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.execute(AsyncExecutionFailureExceptionReportingStage.java:41)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.execute(AsyncExecutionFailureExceptionReportingStage.java:29)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.execute(AsyncApiCallTimeoutTrackingStage.java:64)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallTimeoutTrackingStage.execute(AsyncApiCallTimeoutTrackingStage.java:36)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.execute(AsyncApiCallMetricCollectionStage.java:49)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallMetricCollectionStage.execute(AsyncApiCallMetricCollectionStage.java:32)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.AmazonAsyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonAsyncHttpClient.java:215)
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.invoke(BaseAsyncClientHandler.java:288)
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.doExecute(BaseAsyncClientHandler.java:227)
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.lambda$execute$1(BaseAsyncClientHandler.java:80)
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.measureApiCallSuccess(BaseAsyncClientHandler.java:294)
at software.amazon.awssdk.core.internal.handler.BaseAsyncClientHandler.execute(BaseAsyncClientHandler.java:73)
at software.amazon.awssdk.awscore.client.handler.AwsAsyncClientHandler.execute(AwsAsyncClientHandler.java:49)
at software.amazon.awssdk.services.s3.DefaultS3AsyncClient.headObject(DefaultS3AsyncClient.java:7032)
at software.amazon.awssdk.services.s3.DelegatingS3AsyncClient.lambda$headObject$53(DelegatingS3AsyncClient.java:5259)
at software.amazon.awssdk.services.s3.internal.crossregion.S3CrossRegionAsyncClient.invokeOperation(S3CrossRegionAsyncClient.java:61)
at software.amazon.awssdk.services.s3.DelegatingS3AsyncClient.headObject(DelegatingS3AsyncClient.java:5259)
at software.amazon.awssdk.services.s3.DelegatingS3AsyncClient.lambda$headObject$53(DelegatingS3AsyncClient.java:5259)
at software.amazon.awssdk.services.s3.DelegatingS3AsyncClient.invokeOperation(DelegatingS3AsyncClient.java:10144)
at software.amazon.awssdk.services.s3.DelegatingS3AsyncClient.headObject(DelegatingS3AsyncClient.java:5259)
at software.amazon.s3.analyticsaccelerator.S3SdkObjectClient.headObject(S3SdkObjectClient.java:132)
... 19 more
Caused by: java.lang.IllegalArgumentException: S3Client.makeMetaRequest has invalid options; Operation name must be set for MetaRequestType.DEFAULT.
at software.amazon.awssdk.crt.s3.S3Client.makeMetaRequest(S3Client.java:148)
at software.amazon.awssdk.services.s3.internal.crt.S3CrtAsyncHttpClient.execute(S3CrtAsyncHttpClient.java:163)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.doExecuteHttpRequest(MakeAsyncHttpRequestStage.java:204)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.executeHttpRequest(MakeAsyncHttpRequestStage.java:151)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.lambda$execute$1(MakeAsyncHttpRequestStage.java:104)
at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)
at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683)
at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.execute(MakeAsyncHttpRequestStage.java:100)
at software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage.execute(MakeAsyncHttpRequestStage.java:65)
at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallAttemptMetricCollectionStage.execute(AsyncApiCallAttemptMetricCollectionStage.java:62)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncApiCallAttemptMetricCollectionStage.execute(AsyncApiCallAttemptMetricCollectionStage.java:41)
at software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryingExecutor.attemptExecute(AsyncRetryableStage.java:144)
... 47 more
different error on ITestS3AAnalyticsAcceleratorStreamReading
[ERROR] ITestS3AAnalyticsAcceleratorStreamReading.testMalformedParquetFooter Time elapsed: 2.903 s <<< ERROR!
java.io.IOException: Failed to fetch block data after retries
at software.amazon.s3.analyticsaccelerator.io.physical.data.Block.generateSourceAndData(Block.java:208)
at software.amazon.s3.analyticsaccelerator.io.physical.data.Block.<init>(Block.java:156)
at software.amazon.s3.analyticsaccelerator.io.physical.data.BlockManager.lambda$makeRangeAvailable$1(BlockManager.java:211)
at software.amazon.s3.analyticsaccelerator.common.telemetry.DefaultTelemetry.measureImpl(DefaultTelemetry.java:158)
at software.amazon.s3.analyticsaccelerator.common.telemetry.DefaultTelemetry.measure(DefaultTelemetry.java:79)
at software.amazon.s3.analyticsaccelerator.common.telemetry.Telemetry.measureStandard(Telemetry.java:174)
at software.amazon.s3.analyticsaccelerator.io.physical.data.BlockManager.makeRangeAvailable(BlockManager.java:185)
at software.amazon.s3.analyticsaccelerator.io.physical.data.Blob.read(Blob.java:95)
at software.amazon.s3.analyticsaccelerator.io.physical.impl.PhysicalIOImpl.lambda$read$3(PhysicalIOImpl.java:162)
at software.amazon.s3.analyticsaccelerator.common.telemetry.DefaultTelemetry.measure(DefaultTelemetry.java:102)
at software.amazon.s3.analyticsaccelerator.common.telemetry.Telemetry.measureVerbose(Telemetry.java:253)
at software.amazon.s3.analyticsaccelerator.io.physical.impl.PhysicalIOImpl.read(PhysicalIOImpl.java:151)
at software.amazon.s3.analyticsaccelerator.io.logical.impl.DefaultLogicalIOImpl.lambda$read$1(DefaultLogicalIOImpl.java:92)
at software.amazon.s3.analyticsaccelerator.common.telemetry.DefaultTelemetry.measureConditionally(DefaultTelemetry.java:141)
at software.amazon.s3.analyticsaccelerator.io.logical.impl.DefaultLogicalIOImpl.read(DefaultLogicalIOImpl.java:81)
at software.amazon.s3.analyticsaccelerator.io.logical.impl.ParquetLogicalIOImpl.read(ParquetLogicalIOImpl.java:73)
at software.amazon.s3.analyticsaccelerator.S3SeekableInputStream.lambda$read$3(S3SeekableInputStream.java:155)
at software.amazon.s3.analyticsaccelerator.common.telemetry.DefaultTelemetry.measure(DefaultTelemetry.java:102)
at software.amazon.s3.analyticsaccelerator.common.telemetry.Telemetry.measureVerbose(Telemetry.java:253)
at software.amazon.s3.analyticsaccelerator.S3SeekableInputStream.read(S3SeekableInputStream.java:145)
at org.apache.hadoop.fs.s3a.impl.streams.AnalyticsStream.read(AnalyticsStream.java:123)
at java.io.DataInputStream.read(DataInputStream.java:149)
at org.apache.hadoop.fs.s3a.ITestS3AAnalyticsAcceleratorStreamReading.testMalformedParquetFooter(ITestS3AAnalyticsAcceleratorStreamReading.java:159)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.UnsupportedOperationException: Multipart download is not yet supported. Instead use the CRT based S3 client for multipart download.
at software.amazon.awssdk.services.s3.internal.multipart.MultipartS3AsyncClient.getObject(MultipartS3AsyncClient.java:116)
at software.amazon.s3.analyticsaccelerator.S3SdkObjectClient.getObject(S3SdkObjectClient.java:182)
at software.amazon.s3.analyticsaccelerator.io.physical.data.Block.generateSourceAndData(Block.java:182)
... 37 more
```
|
@steveloughran yeah those errors will go away once I rebase to the latest SDK. will rebase and address your comments now. are you ok with getting this into 3.4.2? i'm just starting the setup work for the release..going through https://github.com/apache/hadoop-release-support/tree/main |
|
If you can address those little nits and and are confident that the errors reported go away, then yes. |
steveloughran
requested changes
Apr 30, 2025
| // region configuration was set to empty string. | ||
| // allow this if people really want it; it is OK to rely on this | ||
| // when deployed in EC2. | ||
| WARN_OF_DEFAULT_REGION_CHAIN.debug(SDK_REGION_CHAIN_IN_USE); |
| region = Region.of(AWS_S3_DEFAULT_REGION); | ||
| builder.withRegion(region); | ||
| origin = "cross region access fallback"; | ||
| } else if (configuredRegion.isEmpty()) { |
There was a problem hiding this comment.
add the option "sdk" here. The full changes I have in mind, including an "ec2" which is only the EC2 side resolution would be a separate patch, but the sdk(which we can leave undocumented) option can ship in this release
There was a problem hiding this comment.
didn't understand, can you clarify, where do i need to add the sdk option?
There was a problem hiding this comment.
imagine we have a special region "sdk" which if set does the same as the empty newline hack
ahmarsuhail
force-pushed
the
HADOOP-19399-crt-client-support
branch
from
ea3ee79 to
66e2b65
Compare
|
rebased and addressed most comments (i think!), will test and double check tomorrow. |
|
💔 -1 overall
This message was automatically generated. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Adds support S3 CRT client.
Configuration mapping b/w clients is getting tricky, I moved endpoint/region resolution logic out so it can be re-used.
CRT does not support client level headers for user agent and requester pays, so those have to be added on a per request level. Only added for
copyObject()andputObject()as those are the only two places CRT will be used in S3A.How was this patch tested?
Tested in
us-east-2withmvn -Dparallel-tests -DtestsThreadCount=16 clean verify. All good, failures isITestS3AEndpointRegionandITestAwsSdkWorkaroundsare related to SDK upgrade.For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?