-
Notifications
You must be signed in to change notification settings - Fork 4.5k
Update iceberg to 1.9.2 #35981
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Update iceberg to 1.9.2 #35981
Changes from all commits
Commits
Show all changes
20 commits
Select commit
Hold shift + click to select a range
7a0051f
update iceberg to 1.9.2
ahmedabu98 b1b4f3e
cleanup
ahmedabu98 15478ba
move iceberg sql into a separate module
ahmedabu98 a92e4e7
check test java version too
ahmedabu98 51c86ff
cleanup
ahmedabu98 308f4f1
spotless
ahmedabu98 bedf6a8
deps
ahmedabu98 5f784ad
add missing dep
ahmedabu98 168c7f5
Merge branch 'master' of https://github.com/ahmedabu98/beam into upda…
ahmedabu98 ca37889
block java 8
ahmedabu98 e04de0c
create examples/iceberg module
ahmedabu98 df7d9d6
dep fixes
ahmedabu98 29fca47
import fixes
ahmedabu98 0da739b
test fix
ahmedabu98 b7da5ce
update changes
ahmedabu98 1de5830
update changes
ahmedabu98 9891e68
include integration tests
ahmedabu98 fde2f93
add examples:iceberg to javaPreCommit
ahmedabu98 08c2d9d
fix deps
ahmedabu98 d9058ff
fix deps
ahmedabu98 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| { | ||
| "comment": "Modify this file in a trivial way to cause this test suite to run.", | ||
| "modification": 3 | ||
| "modification": 4 | ||
| } |
2 changes: 1 addition & 1 deletion
2
.github/trigger_files/IO_Iceberg_Integration_Tests_Dataflow.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| { | ||
| "comment": "Modify this file in a trivial way to cause this test suite to run.", | ||
| "modification": 5 | ||
| "modification": 1 | ||
| } |
2 changes: 1 addition & 1 deletion
2
.github/trigger_files/IO_Iceberg_Managed_Integration_Tests_Dataflow.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| { | ||
| "comment": "Modify this file in a trivial way to cause this test suite to run.", | ||
| "modification": 5 | ||
| "modification": 1 | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| { | ||
| "comment": "Modify this file in a trivial way to cause this test suite to run ", | ||
| "modification": 4 | ||
| "modification": 2 | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * License); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an AS IS BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
|
|
||
| plugins { | ||
| id 'java' | ||
| id 'org.apache.beam.module' | ||
| id 'com.gradleup.shadow' | ||
| } | ||
|
|
||
| applyJavaNature( | ||
| exportJavadoc: false, | ||
| automaticModuleName: 'org.apache.beam.examples.iceberg', | ||
| // iceberg requires Java11+ | ||
| requireJavaVersion: JavaVersion.VERSION_11 | ||
| ) | ||
|
|
||
| description = "Apache Beam :: Examples :: Java :: Iceberg" | ||
| ext.summary = """Apache Beam Java SDK examples using IcebergIO.""" | ||
|
|
||
| /** Define the list of runners which execute a precommit test. | ||
| * Some runners are run from separate projects, see the preCommit task below | ||
| * for details. | ||
| */ | ||
| def preCommitRunners = ["directRunner", "flinkRunner", "sparkRunner"] | ||
| // The following runners have configuration created but not added to preCommit | ||
| def nonPreCommitRunners = ["dataflowRunner", "prismRunner"] | ||
| for (String runner : preCommitRunners) { | ||
| configurations.create(runner + "PreCommit") | ||
| } | ||
| for (String runner: nonPreCommitRunners) { | ||
| configurations.create(runner + "PreCommit") | ||
| } | ||
| configurations.sparkRunnerPreCommit { | ||
| // Ban certain dependencies to prevent a StackOverflow within Spark | ||
| // because JUL -> SLF4J -> JUL, and similarly JDK14 -> SLF4J -> JDK14 | ||
| exclude group: "org.slf4j", module: "jul-to-slf4j" | ||
| exclude group: "org.slf4j", module: "slf4j-jdk14" | ||
| } | ||
|
|
||
| dependencies { | ||
| implementation enforcedPlatform(library.java.google_cloud_platform_libraries_bom) | ||
| runtimeOnly project(":sdks:java:io:iceberg") | ||
| runtimeOnly project(":sdks:java:io:iceberg:bqms") | ||
| implementation project(path: ":sdks:java:core", configuration: "shadow") | ||
| implementation project(":sdks:java:extensions:google-cloud-platform-core") | ||
| implementation project(":sdks:java:io:google-cloud-platform") | ||
| implementation project(":sdks:java:managed") | ||
| implementation library.java.google_auth_library_oauth2_http | ||
| implementation library.java.joda_time | ||
| runtimeOnly project(path: ":runners:direct-java", configuration: "shadow") | ||
| implementation library.java.vendored_guava_32_1_2_jre | ||
| runtimeOnly library.java.hadoop_client | ||
| runtimeOnly library.java.bigdataoss_gcs_connector | ||
|
|
||
| // Add dependencies for the PreCommit configurations | ||
| // For each runner a project level dependency on the examples project. | ||
| for (String runner : preCommitRunners) { | ||
| delegate.add(runner + "PreCommit", project(path: ":examples:java", configuration: "testRuntimeMigration")) | ||
| } | ||
| directRunnerPreCommit project(path: ":runners:direct-java", configuration: "shadow") | ||
| flinkRunnerPreCommit project(":runners:flink:${project.ext.latestFlinkVersion}") | ||
| sparkRunnerPreCommit project(":runners:spark:3") | ||
| sparkRunnerPreCommit project(":sdks:java:io:hadoop-file-system") | ||
| dataflowRunnerPreCommit project(":runners:google-cloud-dataflow-java") | ||
| dataflowRunnerPreCommit project(":runners:google-cloud-dataflow-java:worker") // v2 worker | ||
| dataflowRunnerPreCommit project(":sdks:java:harness") // v2 worker | ||
| prismRunnerPreCommit project(":runners:prism:java") | ||
|
|
||
| // Add dependency if requested on command line for runner | ||
| if (project.hasProperty("runnerDependency")) { | ||
| runtimeOnly project(path: project.getProperty("runnerDependency")) | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| import groovy.json.JsonOutput | ||
|
|
||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * License); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an AS IS BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| plugins { id 'org.apache.beam.module' } | ||
|
|
||
| applyJavaNature( | ||
| automaticModuleName: 'org.apache.beam.sdk.extensions.sql.meta.provider.hcatalog', | ||
| // iceberg requires Java11+ | ||
| requireJavaVersion: JavaVersion.VERSION_11, | ||
| ) | ||
|
|
||
| dependencies { | ||
| implementation project(":sdks:java:extensions:sql") | ||
| implementation project(":sdks:java:core") | ||
| implementation project(":sdks:java:managed") | ||
| implementation project(":sdks:java:io:iceberg") | ||
| runtimeOnly project(":sdks:java:io:iceberg:bqms") | ||
| runtimeOnly project(":sdks:java:io:iceberg:hive") | ||
| // TODO(https://github.com/apache/beam/issues/21156): Determine how to build without this dependency | ||
| provided "org.immutables:value:2.8.8" | ||
| permitUnusedDeclared "org.immutables:value:2.8.8" | ||
| implementation library.java.slf4j_api | ||
| implementation library.java.vendored_guava_32_1_2_jre | ||
| implementation library.java.vendored_calcite_1_40_0 | ||
| implementation library.java.jackson_databind | ||
|
|
||
| testImplementation library.java.joda_time | ||
| testImplementation library.java.junit | ||
| testImplementation library.java.google_api_services_bigquery | ||
| testImplementation "org.apache.iceberg:iceberg-api:1.9.2" | ||
| testImplementation "org.apache.iceberg:iceberg-core:1.9.2" | ||
| testImplementation project(":sdks:java:io:google-cloud-platform") | ||
| testImplementation project(":sdks:java:extensions:google-cloud-platform-core") | ||
| } | ||
|
|
||
| task integrationTest(type: Test) { | ||
| def gcpProject = project.findProperty('gcpProject') ?: 'apache-beam-testing' | ||
| def gcsTempRoot = project.findProperty('gcsTempRoot') ?: 'gs://temp-storage-for-end-to-end-tests/' | ||
|
|
||
| // Disable Gradle cache (it should not be used because the IT's won't run). | ||
| outputs.upToDateWhen { false } | ||
|
|
||
| def pipelineOptions = [ | ||
| "--project=${gcpProject}", | ||
| "--tempLocation=${gcsTempRoot}", | ||
| "--blockOnRun=false"] | ||
|
|
||
| systemProperty "beamTestPipelineOptions", JsonOutput.toJson(pipelineOptions) | ||
|
|
||
| include '**/*IT.class' | ||
|
|
||
| maxParallelForks 4 | ||
| classpath = project(":sdks:java:extensions:sql:iceberg") | ||
| .sourceSets | ||
| .test | ||
| .runtimeClasspath | ||
| testClassesDirs = files(project(":sdks:java:extensions:sql:iceberg").sourceSets.test.output.classesDirs) | ||
| useJUnit { } | ||
| } | ||
|
|
||
| configurations.all { | ||
| // iceberg-core needs avro:1.12.0 | ||
| resolutionStrategy.force 'org.apache.avro:avro:1.12.0' | ||
| } |
File renamed without changes.
31 changes: 31 additions & 0 deletions
31
...ava/org/apache/beam/sdk/extensions/sql/meta/provider/iceberg/IcebergCatalogRegistrar.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| /* | ||
| * Licensed to the Apache Software Foundation (ASF) under one | ||
| * or more contributor license agreements. See the NOTICE file | ||
| * distributed with this work for additional information | ||
| * regarding copyright ownership. The ASF licenses this file | ||
| * to you under the Apache License, Version 2.0 (the | ||
| * "License"); you may not use this file except in compliance | ||
| * with the License. You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
| package org.apache.beam.sdk.extensions.sql.meta.provider.iceberg; | ||
|
|
||
| import com.google.auto.service.AutoService; | ||
| import org.apache.beam.sdk.extensions.sql.meta.catalog.Catalog; | ||
| import org.apache.beam.sdk.extensions.sql.meta.catalog.CatalogRegistrar; | ||
| import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableList; | ||
|
|
||
| @AutoService(CatalogRegistrar.class) | ||
| public class IcebergCatalogRegistrar implements CatalogRegistrar { | ||
| @Override | ||
| public Iterable<Class<? extends Catalog>> getCatalogs() { | ||
| return ImmutableList.<Class<? extends Catalog>>builder().add(IcebergCatalog.class).build(); | ||
| } | ||
| } |
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.