Adding KafkaToGCS #57

YadnikiPawar · 2019-09-25T18:36:16Z

@sabhyankar Pull request for Kafka To GCS Template.

googlebot · 2019-09-25T18:36:24Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

sabhyankar · 2019-09-25T23:55:29Z

@YadnikiPawar - Can you sign the CLA?

YadnikiPawar · 2019-09-26T17:23:49Z

@googlebot I signed it!

googlebot · 2019-09-26T17:23:53Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

sabhyankar

@YadnikiPawar - Thanks for opening this PR. Can you rebase and take a look at the minor nits for now? I want to spend a little more time looking at the main pipeline code.

v2/common/pom.xml

v2/common/src/main/java/com/google/cloud/teleport/v2/coders/FailsafeElementCoder.java

v2/common/src/main/java/com/google/cloud/teleport/v2/coders/package-info.java

v2/common/src/main/java/com/google/cloud/teleport/v2/values/FailsafeElement.java

v2/kafka-to-gcs/README.md

v2/kafka-to-gcs/src/main/resources/kafka-to-gcs-command-spec.json

v2/pom.xml

v2/kafka-to-gcs/src/main/java/com/google/cloud/teleport/v2/templates/KafkaToGCS.java

sabhyankar

I think we need to review this a little more as I disagree with a few of the things.

v2/common/pom.xml

v2/kafka-to-gcs/src/main/java/com/google/cloud/teleport/v2/templates/KafkaToGCS.java

v2/common/src/main/java/com/google/cloud/teleport/v2/transforms/WriteToGCS.java

v2/kafka-to-gcs/src/main/java/com/google/cloud/teleport/v2/templates/KafkaToGCS.java

YadnikiPawar · 2019-11-01T17:07:33Z

@sabhyankar Changes are done. The PR is up for review

YadnikiPawar · 2019-11-05T18:39:53Z

@sabhyankar Removed the unrelated files

sabhyankar

Please add unit tests for each of the WriteToXXX transforms. Also format the classes using the IntelliJ plugin to make it easier.

v2/common/src/main/java/com/google/cloud/teleport/v2/transforms/FileFormatFactory.java

v2/kafka-to-gcs/README.md

v2/kafka-to-gcs/pom.xml

v2/pom.xml

v2/common/src/main/java/com/google/cloud/teleport/v2/transforms/ConstructGenericRecordsFn.java

v2/common/src/main/java/com/google/cloud/teleport/v2/transforms/FileFormatFactory.java

sabhyankar

Minor corrects and nits - Please address and rebase

v2/pom.xml

v2/common/src/main/java/com/google/cloud/teleport/v2/transforms/WriteToGCSAvro.java

v2/common/src/main/java/com/google/cloud/teleport/v2/transforms/WriteToGCSText.java

v2/common/src/main/java/com/google/cloud/teleport/v2/transforms/WriteToGCSUtility.java

v2/common/src/test/java/com/google/cloud/teleport/v2/transforms/WriteToGCSTextTest.java

sabhyankar · 2019-11-08T15:54:37Z

v2/common/src/test/java/com/google/cloud/teleport/v2/transforms/WriteToGCSTextTest.java

Why are we passing null? Should we even use withNumShards if are going to pass null?

@sabhyankar Though of making NumShards a required parameter because when I was trying to set its default value to 0 it threw [java.lang.IllegalArgumentException: When applying WriteFiles to an unbounded PCollection, must specify number of output shards explicitly] this Exception.

@sabhyankar
I made the following changes

Changed withNumShards() to setNumShards().

Setting its default value to be 0 using @Default.Integer(0) in the options Interface

Letting withNumShards() method from IO's (i.e. AvroIO, TextIO, FileIO) to handle this exception

v2/kafka-to-gcs/src/main/java/com/google/cloud/teleport/v2/transforms/FileFormatFactory.java

googlebot · 2019-11-11T08:30:06Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

googlebot · 2019-11-11T13:45:32Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

YadnikiPawar · 2019-11-18T19:47:42Z

@sabhyankar The PR has been rebased.

sabhyankar

LGTM

PiperOrigin-RevId: 281563732

* Dml integration (#53) * Added extensive UT Added extensive UT * Cassandra pr bug fixes (#57) * Cassandra Consolidate Unit Test case and Regression testing fixes (#58) * Added Mapping fixes * Added Spoltles fixes * Added Consolidated fixes * Added TODO * Addess Data and Time * Cassandra pr bug fixes (#64) * Handle TypeHandler Parsing issue fixes (#65) Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> * Added Safe handle (#68) * Handle LocalTime For Time Data Type In Cassandra (#69) * Cassandra pr bug fixes (#70) * Handle Timestamp Fixes (#72) * Added Code Combined in a single way * Address The Unwanted Hop * Cassandra pr bug fixes (#75) * Added PR Review Comments * Remove NamesCol Dependecy as spannerTableName is same as In Given Mapping * Added spannerTableId for fetching Mapping * Removed SpannerToID and also Updated Session file with proper structure * Timestamp in milisecond * removed assertNotNull from UT wherever possible * Added Fixes * Added Note Instead of Question * -- review fixes (#78) * Added Bytes to hex to blob conversion * Handling Bytes as Binary encoded As of now * Passing Null Value to Primary Key as well for cassandra * Added UT fixes * Added UT refectoring * Reverse merge confict fixes --------- Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com> Co-authored-by: Akash Thawait <aakash@ollion.com>

googlebot added the cla: no The PR submitter does not have a CLA label Sep 25, 2019

sabhyankar self-requested a review September 25, 2019 23:55

googlebot added cla: yes The PR submitter has a CLA and removed cla: no The PR submitter does not have a CLA labels Sep 26, 2019

YadnikiPawar force-pushed the master branch 3 times, most recently from 586a115 to 7375a66 Compare October 2, 2019 20:25

sabhyankar requested changes Oct 2, 2019

View reviewed changes

YadnikiPawar force-pushed the master branch 4 times, most recently from 3ef5f58 to 401ab72 Compare October 2, 2019 21:51

sabhyankar requested changes Oct 17, 2019

View reviewed changes

YadnikiPawar closed this Oct 30, 2019

YadnikiPawar force-pushed the master branch from 27d37f0 to 8fa65eb Compare October 30, 2019 22:00

YadnikiPawar reopened this Oct 31, 2019

YadnikiPawar closed this Nov 5, 2019

YadnikiPawar force-pushed the master branch from 9ac34d2 to 8fa65eb Compare November 5, 2019 17:54

YadnikiPawar reopened this Nov 5, 2019

sabhyankar requested changes Nov 6, 2019

View reviewed changes

YadnikiPawar force-pushed the master branch from dc3f8be to 53181ba Compare November 8, 2019 11:18

YadnikiPawar requested a review from sabhyankar November 8, 2019 11:22

sabhyankar requested changes Nov 8, 2019

View reviewed changes

YadnikiPawar force-pushed the master branch from 314bfcb to f69f8f4 Compare November 9, 2019 10:05

YadnikiPawar requested a review from sabhyankar November 9, 2019 10:11

YadnikiPawar force-pushed the master branch 2 times, most recently from 17959ab to bfb196b Compare November 11, 2019 08:30

googlebot added cla: no The PR submitter does not have a CLA and removed cla: yes The PR submitter has a CLA labels Nov 11, 2019

YadnikiPawar closed this Nov 11, 2019

YadnikiPawar force-pushed the master branch from fb22272 to 289c5ba Compare November 11, 2019 12:58

YadnikiPawar reopened this Nov 11, 2019

googlebot added cla: yes The PR submitter has a CLA and removed cla: no The PR submitter does not have a CLA labels Nov 11, 2019

YadnikiPawar force-pushed the master branch from c205c04 to 62e63cf Compare November 12, 2019 09:21

Adding KafkaToGCS Refactored

ace7a40

YadnikiPawar force-pushed the master branch from 62e63cf to ace7a40 Compare November 18, 2019 19:41

sabhyankar approved these changes Nov 18, 2019

View reviewed changes

sabhyankar added the Google LGTM Approval of a pull request to be merged into the repository label Nov 18, 2019

ryanmcdowell pushed a commit that referenced this pull request Nov 20, 2019

Merge pull request #57 from YadnikiPawar:master

54a4181

PiperOrigin-RevId: 281563732

ryanmcdowell merged commit ace7a40 into GoogleCloudPlatform:master Nov 20, 2019

Adding KafkaToGCS #57

Adding KafkaToGCS #57

Uh oh!

Conversation

YadnikiPawar commented Sep 25, 2019

Uh oh!

googlebot commented Sep 25, 2019

What to do if you already signed the CLA

Individual signers

Corporate signers

Uh oh!

sabhyankar commented Sep 25, 2019

Uh oh!

YadnikiPawar commented Sep 26, 2019

Uh oh!

googlebot commented Sep 26, 2019

Uh oh!

sabhyankar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sabhyankar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

YadnikiPawar commented Nov 1, 2019

Uh oh!

YadnikiPawar commented Nov 5, 2019

Uh oh!

sabhyankar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sabhyankar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sabhyankar Nov 8, 2019

Choose a reason for hiding this comment

Uh oh!

YadnikiPawar Nov 9, 2019

Choose a reason for hiding this comment

Uh oh!

YadnikiPawar Nov 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

YadnikiPawar Nov 9, 2019 •

edited

Loading