Skip to content

Conversation

@lagarwal1
Copy link
Contributor

@lagarwal1 lagarwal1 commented Jan 23, 2025

feat: Add support for importing/exporting UUID columns as strings

Manual Tests in Cloud Devel

Avro

GSQL

Export
Import

Postgres

Export
Import

CSV

GSQL

Export
Import

Postgres

Export
Import

@lagarwal1 lagarwal1 requested a review from a team as a code owner January 23, 2025 05:16
@darshan-sj darshan-sj added the improvement Making existing code better label Jan 23, 2025
@codecov
Copy link

codecov bot commented Jan 23, 2025

Codecov Report

Attention: Patch coverage is 58.02469% with 34 lines in your changes missing coverage. Please review.

Project coverage is 46.95%. Comparing base (dae14d7) to head (4ca86eb).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
...e/beam/it/gcp/spanner/matchers/SpannerAsserts.java 0.00% 13 Missing ⚠️
...d/teleport/templates/common/SpannerConverters.java 0.00% 4 Missing and 4 partials ⚠️
...oud/teleport/spanner/AvroSchemaToDdlConverter.java 20.00% 3 Missing and 1 partial ⚠️
...le/cloud/teleport/spanner/TextImportTransform.java 0.00% 2 Missing and 2 partials ⚠️
...google/cloud/teleport/spanner/ExportTransform.java 0.00% 2 Missing ⚠️
...cloud/teleport/spanner/SpannerRecordConverter.java 77.77% 0 Missing and 2 partials ⚠️
...le/cloud/teleport/spanner/CSVRecordToMutation.java 50.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##               main    #2145    +/-   ##
==========================================
  Coverage     46.95%   46.95%            
- Complexity     4043     4367   +324     
==========================================
  Files           876      876            
  Lines         52154    52190    +36     
  Branches       5477     5501    +24     
==========================================
+ Hits          24489    24507    +18     
- Misses        25912    25923    +11     
- Partials       1753     1760     +7     
Components Coverage Δ
spanner-templates 68.88% <65.57%> (-0.01%) ⬇️
spanner-import-export 65.70% <65.57%> (-0.02%) ⬇️
spanner-live-forward-migration 76.50% <ø> (ø)
spanner-live-reverse-replication 78.67% <ø> (ø)
spanner-bulk-migration 87.94% <ø> (ø)
Files with missing lines Coverage Δ
...le/cloud/teleport/spanner/AvroRecordConverter.java 99.69% <ø> (ø)
...teleport/spanner/BuildReadFromTableOperations.java 44.92% <100.00%> (+9.33%) ⬆️
...oud/teleport/spanner/DdlToAvroSchemaConverter.java 93.05% <100.00%> (+0.02%) ⬆️
...gle/cloud/teleport/spanner/TextImportPipeline.java 0.00% <ø> (ø)
...oogle/cloud/teleport/spanner/common/SizedType.java 92.10% <100.00%> (+0.25%) ⬆️
...com/google/cloud/teleport/spanner/common/Type.java 90.74% <100.00%> (+0.42%) ⬆️
.../com/google/cloud/teleport/spanner/ddl/Column.java 84.95% <100.00%> (+0.55%) ⬆️
...google/cloud/teleport/templates/TextToSpanner.java 23.43% <100.00%> (+0.40%) ⬆️
.../apache/beam/sdk/io/gcp/spanner/SpannerSchema.java 67.25% <100.00%> (+1.20%) ⬆️
...le/cloud/teleport/spanner/CSVRecordToMutation.java 79.31% <50.00%> (-0.52%) ⬇️
... and 6 more

... and 2 files with indirect coverage changes

@lagarwal1 lagarwal1 changed the title Add UUID support in import/export templates [WIP] Add UUID support in import/export templates Jan 23, 2025
@lagarwal1 lagarwal1 marked this pull request as draft January 23, 2025 08:58
@lagarwal1 lagarwal1 marked this pull request as ready for review January 29, 2025 17:01
@lagarwal1 lagarwal1 changed the title [WIP] Add UUID support in import/export templates Add UUID support in import/export templates Jan 29, 2025
darshan-sj
darshan-sj previously approved these changes Feb 7, 2025
Copy link
Contributor

@darshan-sj darshan-sj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall. Left a few comments.

Copy link
Contributor

@darshan-sj darshan-sj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@darshan-sj
Copy link
Contributor

Unrelated test failure in Spanner PR / Dataflow Templates Integration Tests (pull_request): com.google.cloud.teleport.v2.templates.SpannerToSourceDbCustomTransformationIT

@darshan-sj darshan-sj merged commit becc7f8 into GoogleCloudPlatform:main Feb 10, 2025
27 of 28 checks passed
@Abacn
Copy link
Contributor

Abacn commented Feb 13, 2025

Please sync the change to Beam's SpannerIO : https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerSchema.java

LocalSpannerIO is going to be teardown

@Abacn
Copy link
Contributor

Abacn commented Feb 14, 2025

LocalSpannerIO does not admit changes that not already in Beam's SpannerIO, and is going to be teardown

#1429 will affect part of the functionality added in this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

improvement Making existing code better size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants