Skip to content

(WIP) add support for throttling bulk writes #160

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
aananthram wants to merge 2 commits into from
Closed

(WIP) add support for throttling bulk writes #160

aananthram wants to merge 2 commits into from

Conversation

@aananthram
Copy link

@aananthram aananthram commented Aug 17, 2020

No description provided.

@google-cla
Copy link

google-cla bot commented Aug 17, 2020

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

@google-cla google-cla bot added the cla: no The PR submitter does not have a CLA label Aug 17, 2020
@aananthram
Copy link
Author

aananthram commented Aug 17, 2020

@googlebot I signed it!

@google-cla
Copy link

google-cla bot commented Aug 17, 2020

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers
Corporate signers

ℹ️ Googlers: Go here for more info.

sabhyankar
sabhyankar reviewed Aug 17, 2020
View reviewed changes
src/main/java/com/google/cloud/teleport/bigtable/AvroToBigtable.java
Comment on lines +118 to +122
BulkOptions.Builder bulkOptionsBuilder = BulkOptions.newBuilder();
if (options.getEnableBulkMutationThrottling() != null && options.getEnableBulkMutationThrottling().get()) {
bulkOptionsBuilder.enableBulkMutationThrottling();
}

Copy link
Member

@sabhyankar sabhyankar Aug 17, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @aananthram - This part runs at graph creation time and not at runtime so whether we enableBulkMutationThrottling will be determined at the template creation and is not going at runtime.

Copy link
Author

@aananthram aananthram Aug 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for pointing that out! how are the args for project, etc, handled below at runtime then?

Copy link
Member

@sabhyankar sabhyankar Aug 31, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at our SDK, it seems the projectid and instanceid should not be provided[1]

"WARNING: instanceId and projectId should not be provided here and should be provided over withProjectId(org.apache.beam.sdk.options.ValueProvider<java.lang.String>) and withInstanceId(org.apache.beam.sdk.options.ValueProvider<java.lang.String>)."

[1] https://beam.apache.org/releases/javadoc/2.23.0/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.Write.html#withBigtableOptionsConfigurator-org.apache.beam.sdk.transforms.SerializableFunction-

Copy link
Author

@aananthram aananthram Sep 15, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, i see. so that allows them to be provided at runtime. is there a way to enable bulk mutation at runtime then?

@google-cla google-cla bot added cla: yes The PR submitter has a CLA and removed cla: no The PR submitter does not have a CLA labels Aug 17, 2020
sabhyankar
sabhyankar requested changes Aug 31, 2020
View reviewed changes
src/main/java/com/google/cloud/teleport/bigtable/AvroToBigtable.java
Comment on lines +118 to +122
BulkOptions.Builder bulkOptionsBuilder = BulkOptions.newBuilder();
if (options.getEnableBulkMutationThrottling() != null && options.getEnableBulkMutationThrottling().get()) {
bulkOptionsBuilder.enableBulkMutationThrottling();
}

Copy link
Member

@sabhyankar sabhyankar Aug 31, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at our SDK, it seems the projectid and instanceid should not be provided[1]

"WARNING: instanceId and projectId should not be provided here and should be provided over withProjectId(org.apache.beam.sdk.options.ValueProvider<java.lang.String>) and withInstanceId(org.apache.beam.sdk.options.ValueProvider<java.lang.String>)."

[1] https://beam.apache.org/releases/javadoc/2.23.0/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.Write.html#withBigtableOptionsConfigurator-org.apache.beam.sdk.transforms.SerializableFunction-

@stale
Copy link

stale bot commented Oct 31, 2020

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the obsolete The PR hasn't had activity in 45 days label Oct 31, 2020
@stale stale bot closed this Nov 7, 2020
VardhanThigle added a commit that referenced this pull request Apr 3, 2025
@akashthawaitcc @taherkl
@pawankashyapollion @VardhanThigle @asthamohta @jrmccluskey @Abacn @jjfox15 @vgnanasekaran @damccorm @shreyakhajanchi @Rudra-Gujarathi @derrickaw @darshan-sj @svetakvsundhar @shunping @dedocibula
Bug fixes of handling null values and added missing IT (#2268)
4f920fb 
* Acceptance Test Fixes

* sync upstream/main (#98)

* Enhanced Retry Logic (#2196)

Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com>

* Adding support for Cassandra map (#2209)

* Adding test table for Map types

* Adding support for cassandra map

* changes (#2212)

* Fix inf issues in Datastream reader (#2213)

* add inf fix to ts

fixing bug on timestamp type

* Update FormatDatastreamRecordToJson.java

use equals not ==

* Upgrade Beam version to 2.63.0 (#2206)

* Upgrade Beam version to 2.63.0

* add cache factory to local spanner io

* remove cache pass to ReadChangeStreamPartitionDoFn

* Add unimplemented/unused stubs to TestChangeStreamMutation

* Use java Instant instead of threeten Instant

* Fix low watermark setter call

* fix testchangestreammutation

* Recreate validation PR

* fix import order

* linux-env requirements files

* fix v1 pom

* SkipShade for Spanner common module (#2194)

* Add load test for cross db txn (#2199)

* Add load test for cross db txn

* Change test timeout to 2 days

* Revert spanner-pr.yml changes

* Report Lineage for CsvToBigQuery template (#2205)

* Report Lineage for CsvToBigQuery template

* Reply beampr-32662 to CsvConverters

* Spanner Import/Export INTERLEAVE IN (#2128)

* Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet.

* Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set.

* Set interleaveType in InfoSchemaScanner

* Style fixes, and only generate INTERLEAVE IN ddl for gsql.

* another style fix - remove unused import

* Make conditions more readable and add comments

* Tests

* Fix condition

* Add interleave in table to ExportPipelineIT

* Add SCRAM-SHA-512 authentication support to Kafka templates (#2181)

Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates.

* Update the required Java version in the base doc, then regenerate docs (#2218)

* Update java requirement

* Generate docs

* Post 2.63.0 fixes (#2216)

* Uncomment kinesis

* bump protoc to 4 in v2

* bug-fix: Use jdbc connection properties for reverse migration (#2198)

* changes

* Changes

* changes

* changes

* docs

* Support partitioned reads for DateTime column type in JDBC to BigQuery template (#2084)

* Support partitioned reads for DateTime column type

* minor changes

* Support backward compatibility and timezone in lower/upper bounds

* removed test cases for bounds in unit tests as derby does not support timezone

* Fixed typo's in JdbcToBigQuery.java
Corrected testcases expected output

* Corrected unit test cases expected output. Removed DateTime integration test cases.

* Corrected a typo

* Added default value for partitionColumnType

* spotless apply

* Disabling flaky test to unblock dataflow release (#2220)

* disabling flaky test to unblock dataflow release

* minor change

* Add logic to skip runnerV2 for the ITs (#2219)

* skip runnerv2 tests in TemplateTestBase

* add logic to skip use_runner_v2 experiment in the launchTemplate

* Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate

* Fix format violations using mvn spotless:apply

* Add warning about caching with plugin (#2221)

* [DatastreamToSpanner] Spanner Exception handling (#2185)

* SpannerExceptionClassifier class and IT

* Unit tests and Integration tests

* Formatting changes

* Correcting UT

* Unit test for SpannerResourceManager

* Creating Spanner Migration Exception

* Formatting changes

* Correcting tests

* Addressing comments

* Correcting a UT

* Addressing comments

* Changing SpannerMigrationException to extend RuntimeException

* Remove Python version from `pom.xml` (#2234)

* Update pom.xml

* Update pom.xml

* Add SkipRunnerV2Test category to JmsToPubsubIT and PubSubCdcToBigQueryIT (#2235)

* Fix a bug in CSVToBigQuery where commas in fields are not handled correctly. (#2229)

* Attempt to fix csv bug where commas in fields are not handled correctly.

* Replace ImmutableList with Iterable

* Add tests to cover the scenario of commas within quotes.

* Polish tests and add a test case to cover csv without headers.

* Update Dockerfile-template-yaml (#2222)

* Update Dockerfile-template-yaml

* Update Dockerfile-template-yaml

* Update Dockerfile-template-yaml

* Update Dockerfile-template-yaml

* Adding All Datatypes IT for Cassandra Migration (#2230)

* Add IF NOT EXISTS clause for spanner ddls used in ITs (#2237)

* Enable DatastreamToSpannerIT with if not exists clause

* Update Datastream to Spanner IT spanner schemas with if not exists

* Update BULK IT spanner schemas with if not exists

* Update reverse replications ITs spanner schemas with if not exists

* Add space after if not exists

* Using set of random buckets for spanner ITs (#2223)

* Using set of random buckets for spanner ITs

* reverse replication test

* checkstyle fix

* adding more buckets

* removed ignore for testing

* spotless fix

* fixing UT

* skip the flaky test again

* addressing comments

* spotless fix

* Add promote artifact method in release plugin (#2227)

* Add promote artifact method

* address comments; also fixed stagingArtifactRegistry support us.gcr.io

* Consolidate method

* fixed default DLQ path (#2241)

* Fix stagingArtifactRegistry support raw us.gcr.io artifact registry (#2243)

* Print error response on wget call (#2245)

* This helps debugging e.g. permission issues

* Moving local spanner io to a different namespace (#2231)

* Removed LocalSpannerIO

* Moving all of SpannerIO into Teleport

* Copying tests also

* Updated tests and excluded SpannerIO changestream from coverage checks

* spotless apply

* Excluding coverage check

* Adding retry settings which were overwritten in LocalSpannerAccessor

* Changing deprecated retrysettings function calls

* Added warning and TODO comments to remove the local copy

* spotless

* Adding Cassandra Type Options to IT test (#2242)

* Bump timeouts for tests involving FKs/interleaved dependenceis (#2239)

* Bump timeouts for datastream to spanner test

* Update FK timeout for reverse template

* Bump timeouts for old reverse repl template

* Enable SpannerToSourceDbInterleaveMultiShardIT

* Revert loadtest timeout

* Load Tests - Cassandra Reverse Replication (#2163)

* * Addition of Load Tests in SpannerToSourceDB For Cassandra (#89)

* Addition of Load Tests in SpannerToSourceDB For Cassandra

* Address Merge conflict

* Added LT Refectored (#92)

* Added POM Dependecies

* sync upstream/main (#98)

* RR LOAD TEST FIXES (#101)

* Resolved PR comments (#115)

* Added Module Dependency Fixes

* Added Copyrigh

* Added missing commit

* Enhanced Retry Logic (#2196)

Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com>

* Adding support for Cassandra map (#2209)

* Adding test table for Map types

* Adding support for cassandra map

* changes (#2212)

* Fix inf issues in Datastream reader (#2213)

* add inf fix to ts

fixing bug on timestamp type

* Update FormatDatastreamRecordToJson.java

use equals not ==

* Upgrade Beam version to 2.63.0 (#2206)

* Upgrade Beam version to 2.63.0

* add cache factory to local spanner io

* remove cache pass to ReadChangeStreamPartitionDoFn

* Add unimplemented/unused stubs to TestChangeStreamMutation

* Use java Instant instead of threeten Instant

* Fix low watermark setter call

* fix testchangestreammutation

* Recreate validation PR

* fix import order

* linux-env requirements files

* fix v1 pom

* SkipShade for Spanner common module (#2194)

* Add load test for cross db txn (#2199)

* Add load test for cross db txn

* Change test timeout to 2 days

* Revert spanner-pr.yml changes

* Report Lineage for CsvToBigQuery template (#2205)

* Report Lineage for CsvToBigQuery template

* Reply beampr-32662 to CsvConverters

* Spanner Import/Export INTERLEAVE IN (#2128)

* Changes to write read interleave type from information schema, and write/read the type to/from avro. No change to tests yet.

* Fix warnings, bug in InformationSchemaScanner, and bug in AvroSchemaToDdlConverter. Also properly default to IN PARENT when emitting ddl, in case the interleave type is not set (really only necessary for tests, since otherwise it will always be set.

* Set interleaveType in InfoSchemaScanner

* Style fixes, and only generate INTERLEAVE IN ddl for gsql.

* another style fix - remove unused import

* Make conditions more readable and add comments

* Tests

* Fix condition

* Add interleave in table to ExportPipelineIT

* Add SCRAM-SHA-512 authentication support to Kafka templates (#2181)

Added SCRAM-SHA-512 authentication support to Kafka to Kafka, Kafka to GCS, and Kafka to BigQuery templates.

* Update the required Java version in the base doc, then regenerate docs (#2218)

* Update java requirement

* Generate docs

* Post 2.63.0 fixes (#2216)

* Uncomment kinesis

* bump protoc to 4 in v2

* bug-fix: Use jdbc connection properties for reverse migration (#2198)

* changes

* Changes

* changes

* changes

* docs

* Support partitioned reads for DateTime column type in JDBC to BigQuery template (#2084)

* Support partitioned reads for DateTime column type

* minor changes

* Support backward compatibility and timezone in lower/upper bounds

* removed test cases for bounds in unit tests as derby does not support timezone

* Fixed typo's in JdbcToBigQuery.java
Corrected testcases expected output

* Corrected unit test cases expected output. Removed DateTime integration test cases.

* Corrected a typo

* Added default value for partitionColumnType

* spotless apply

* Disabling flaky test to unblock dataflow release (#2220)

* disabling flaky test to unblock dataflow release

* minor change

* Add logic to skip runnerV2 for the ITs (#2219)

* skip runnerv2 tests in TemplateTestBase

* add logic to skip use_runner_v2 experiment in the launchTemplate

* Remove redundant property skipRunnerV2Test in pom and update logic in launchTemplate

* Fix format violations using mvn spotless:apply

* Add warning about caching with plugin (#2221)

* [DatastreamToSpanner] Spanner Exception handling (#2185)

* SpannerExceptionClassifier class and IT

* Unit tests and Integration tests

* Formatting changes

* Correcting UT

* Unit test for SpannerResourceManager

* Creating Spanner Migration Exception

* Formatting changes

* Correcting tests

* Addressing comments

* Correcting a UT

* Addressing comments

* Changing SpannerMigrationException to extend RuntimeException

* Added Cassandra Resource Manager Refectoring and removed Generics

* Added Keyspace Voilation fixes

* minor changes

* Create session for row check

---------

Co-authored-by: taherkl <taher.lakdawala@ollion.com>
Co-authored-by: Akash Thawait <aakash@ollion.com>
Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com>
Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com>
Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com>
Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com>
Co-authored-by: Yi Hu <yathu@google.com>
Co-authored-by: jjfox15 <seanfox@google.com>
Co-authored-by: vgnanasekaran <venku123@gmail.com>
Co-authored-by: Danny McCormick <dannymccormick@google.com>
Co-authored-by: shreyakhajanchi <92910380+shreyakhajanchi@users.noreply.github.com>
Co-authored-by: Rudra-Gujarathi <rudragujarathi@google.com>
Co-authored-by: Derrick Williams <derrickaw@google.com>
Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com>

* Added Missing Case

* Added IT for Delete

* Added For testing

* Added IT for Emty and Custom handling

* fixed cassandra schema

* Added And removed unwanted Table

* Added And removed unwanted

* Added Missing case and removed Delete case

* Spoltless fixes

* Handle Null in custom Transformation

* revert delete changes

* Handle Checkstyle fixes

* removed unwanted toSting

---------

Co-authored-by: taherkl <taher.lakdawala@ollion.com>
Co-authored-by: Taher Lakdawala <78196491+taherkl@users.noreply.github.com>
Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com>
Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com>
Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com>
Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com>
Co-authored-by: Yi Hu <yathu@google.com>
Co-authored-by: jjfox15 <seanfox@google.com>
Co-authored-by: vgnanasekaran <venku123@gmail.com>
Co-authored-by: Danny McCormick <dannymccormick@google.com>
Co-authored-by: shreyakhajanchi <92910380+shreyakhajanchi@users.noreply.github.com>
Co-authored-by: Rudra-Gujarathi <rudragujarathi@google.com>
Co-authored-by: Derrick Williams <derrickaw@google.com>
Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com>
Co-authored-by: Svetak Sundhar <svetaksundhar@google.com>
Co-authored-by: Shunping Huang <shunping@google.com>
Co-authored-by: Andrej Galad <dedo.cibula@gmail.com>

* ADDED Missing Combination (#160)

* Adding missing col combination assert for empty (#161)

* ADDED Missing Combination

* Address the assert the same

* Schema Fixes

* Schema FIxes

* Added SCHEMA FIXES

* Schema Correction

* Added Fixes

* Added Schema Fixes

* Adding InsertOnly Mode for Bulk job (#2260)

* Adding Lgs

* Handle Null Explicit

* removed logs

* revert ignore change

* revert additional changes

* Added UT for Null and Empty Case (#171)

---------

Co-authored-by: taherkl <taher.lakdawala@ollion.com>
Co-authored-by: Taher Lakdawala <78196491+taherkl@users.noreply.github.com>
Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com>
Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com>
Co-authored-by: Astha Mohta <35952883+asthamohta@users.noreply.github.com>
Co-authored-by: Jack McCluskey <34928439+jrmccluskey@users.noreply.github.com>
Co-authored-by: Yi Hu <yathu@google.com>
Co-authored-by: jjfox15 <seanfox@google.com>
Co-authored-by: vgnanasekaran <venku123@gmail.com>
Co-authored-by: Danny McCormick <dannymccormick@google.com>
Co-authored-by: shreyakhajanchi <92910380+shreyakhajanchi@users.noreply.github.com>
Co-authored-by: Rudra-Gujarathi <rudragujarathi@google.com>
Co-authored-by: Derrick Williams <derrickaw@google.com>
Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com>
Co-authored-by: Svetak Sundhar <svetaksundhar@google.com>
Co-authored-by: Shunping Huang <shunping@google.com>
Co-authored-by: Andrej Galad <dedo.cibula@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sabhyankar sabhyankar sabhyankar requested changes

Assignees

No one assigned

Labels

cla: yes The PR submitter has a CLA obsolete The PR hasn't had activity in 45 days

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@aananthram @sabhyankar