Skip to content

HADOOP-19480. Upgrades AAL version to 1.0.0. #7469

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 6, 2025
Merged

HADOOP-19480. Upgrades AAL version to 1.0.0. #7469

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
425edc0
upgrades AAL version to 1.0.0
ahmarsuhail Mar 5, 2025
7492d1e
updates dependency in hadoop-project
ahmarsuhail Mar 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion hadoop-project/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@
<aws-java-sdk.version>1.12.720</aws-java-sdk.version>
<aws-java-sdk-v2.version>2.25.53</aws-java-sdk-v2.version>
<amazon-s3-encryption-client-java.version>3.1.1</amazon-s3-encryption-client-java.version>
<amazon-s3-analyticsaccelerator-s3.version>0.0.4</amazon-s3-analyticsaccelerator-s3.version>
<amazon-s3-analyticsaccelerator-s3.version>1.0.0</amazon-s3-analyticsaccelerator-s3.version>
<aws.eventstream.version>1.0.1</aws.eventstream.version>
<hsqldb.version>2.7.1</hsqldb.version>
<frontend-maven-plugin.version>1.11.2</frontend-maven-plugin.version>
Expand Down
32 changes: 0 additions & 32 deletions ...aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3AAnalyticsAcceleratorStreamReading.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,6 @@

import software.amazon.s3.analyticsaccelerator.S3SeekableInputStreamConfiguration;
import software.amazon.s3.analyticsaccelerator.common.ConnectorConfiguration;
import software.amazon.s3.analyticsaccelerator.util.PrefetchMode;

import static org.apache.hadoop.fs.Options.OpenFileOptions.FS_OPTION_OPENFILE_READ_POLICY;
import static org.apache.hadoop.fs.Options.OpenFileOptions.FS_OPTION_OPENFILE_READ_POLICY_PARQUET;
Expand All @@ -64,8 +63,6 @@
public class ITestS3AAnalyticsAcceleratorStreamReading extends AbstractS3ATestBase {

private static final String PHYSICAL_IO_PREFIX = "physicalio";
Copy link

@fuatbasik fuatbasik Mar 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can get rid of this private static final String PHYSICAL_IO_PREFIX = "physicalio"; too right?

Copy link
Contributor Author

@ahmarsuhail ahmarsuhail Mar 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no it's used in the last test

private static final String LOGICAL_IO_PREFIX = "logicalio";


private Path externalTestFile;

Expand Down Expand Up @@ -176,35 +173,6 @@ public void testMultiRowGroupParquet() throws Throwable {
verifyStatisticCounterValue(ioStats, STREAM_READ_ANALYTICS_OPENED, 1);
}

@Test
public void testConnectorFrameworkConfigurable() {
describe("Verify S3 connector framework reads configuration");

Configuration conf = new Configuration(getConfiguration());

//Disable Predictive Prefetching
conf.set(ANALYTICS_ACCELERATOR_CONFIGURATION_PREFIX +
"." + LOGICAL_IO_PREFIX + ".prefetching.mode", "all");

//Set Blobstore Capacity
conf.setInt(ANALYTICS_ACCELERATOR_CONFIGURATION_PREFIX +
"." + PHYSICAL_IO_PREFIX + ".blobstore.capacity", 1);

ConnectorConfiguration connectorConfiguration =
new ConnectorConfiguration(conf, ANALYTICS_ACCELERATOR_CONFIGURATION_PREFIX);

S3SeekableInputStreamConfiguration configuration =
S3SeekableInputStreamConfiguration.fromConfiguration(connectorConfiguration);

Assertions.assertThat(configuration.getLogicalIOConfiguration().getPrefetchingMode())
.as("AnalyticsStream configuration is not set to expected value")
.isSameAs(PrefetchMode.ALL);

Assertions.assertThat(configuration.getPhysicalIOConfiguration().getBlobStoreCapacity())
.as("AnalyticsStream configuration is not set to expected value")
.isEqualTo(1);
}

@Test
public void testInvalidConfigurationThrows() throws Exception {
describe("Verify S3 connector framework throws with invalid configuration");
Expand Down