Skip to content

Conversation

@VardhanThigle
Copy link
Contributor

@VardhanThigle VardhanThigle commented Dec 4, 2024

Cassandra Schema and Data mapping for primitive types.

This PR covers mapping of primitive types to Avro for Cassandra.
As of this PR, all the non-primitive types are mapped to null.

TODOS:

  • end to end integration with pipeline options and CassandraIO.
  • integration and scale testing
  • support for non-primitive types.

@codecov
Copy link

codecov bot commented Dec 4, 2024

Codecov Report

Attention: Patch coverage is 99.40828% with 1 line in your changes missing coverage. Please review.

Project coverage is 46.16%. Comparing base (ff60a5f) to head (b9af015).

Files with missing lines Patch % Lines
.../cassandra/mappings/CassandraMappingsProvider.java 98.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2048      +/-   ##
============================================
+ Coverage     45.98%   46.16%   +0.17%     
- Complexity     3812     4167     +355     
============================================
  Files           851      856       +5     
  Lines         50844    51010     +166     
  Branches       5359     5365       +6     
============================================
+ Hits          23383    23547     +164     
- Misses        25759    25760       +1     
- Partials       1702     1703       +1     
Components Coverage Δ
spanner-templates 67.89% <99.32%> (+0.27%) ⬆️
spanner-import-export 65.57% <ø> (-0.02%) ⬇️
spanner-live-forward-migration 76.42% <100.00%> (+0.11%) ⬆️
spanner-live-reverse-replication 77.35% <100.00%> (+0.09%) ⬆️
spanner-bulk-migration 87.16% <99.32%> (+0.29%) ⬆️
Files with missing lines Coverage Δ
...tream/transforms/FormatDatastreamRecordToJson.java 42.58% <100.00%> (+3.95%) ⬆️
...eader/io/cassandra/mappings/CassandraMappings.java 100.00% <100.00%> (ø)
...r/io/cassandra/rowmapper/CassandraFieldMapper.java 100.00% <100.00%> (ø)
...der/io/cassandra/rowmapper/CassandraRowMapper.java 100.00% <100.00%> (ø)
.../cassandra/rowmapper/CassandraSourceRowMapper.java 100.00% <100.00%> (ø)
.../io/cassandra/schema/CassandraSchemaDiscovery.java 98.24% <ø> (ø)
...eader/io/schema/typemapping/UnifiedTypeMapper.java 100.00% <100.00%> (ø)
...ema/typemapping/provider/unified/CustomSchema.java 100.00% <100.00%> (ø)
...pping/provider/unified/UnifiedMappingProvider.java 100.00% <100.00%> (ø)
...er/migrations/avro/GenericRecordTypeConvertor.java 93.58% <100.00%> (+0.46%) ⬆️
... and 1 more

... and 1 file with indirect coverage changes

@VardhanThigle VardhanThigle force-pushed the cassandra-mapping branch 9 times, most recently from 9a82e35 to 3d23485 Compare December 11, 2024 10:03
@VardhanThigle VardhanThigle force-pushed the cassandra-mapping branch 6 times, most recently from 46039b0 to 452f060 Compare December 12, 2024 04:53
@VardhanThigle VardhanThigle marked this pull request as ready for review December 12, 2024 04:53
@VardhanThigle VardhanThigle requested a review from a team as a code owner December 12, 2024 04:53
@VardhanThigle VardhanThigle force-pushed the cassandra-mapping branch 2 times, most recently from 8118ab4 to 677859f Compare December 12, 2024 08:45
@VardhanThigle VardhanThigle requested a review from a team as a code owner December 12, 2024 08:45
@Deep1998
Copy link
Contributor

LGTM

Copy link
Contributor

@bharadwaj-aditya bharadwaj-aditya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks fine overall. Please address the comments before merging.

@VardhanThigle VardhanThigle force-pushed the cassandra-mapping branch 3 times, most recently from 37a95ec to b9af015 Compare December 13, 2024 04:31
Copy link
Contributor Author

@VardhanThigle VardhanThigle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IT tests are failing, due to

{"errors":[{"code":"TOOMANYREQUESTS","message":"Quota exceeded for quota metric 'Requests per project in the US multi-region' and limit 'Requests per project in the US multi-region per minute' of service 'artifactregistry.googleapis.com' for consumer 'project_number:167633336322'."}]}

The same workflow had passed on this PR 13 hours ago.

Even the scheduled workflow on main seems to be flaky.

@VardhanThigle VardhanThigle merged commit cd28d01 into GoogleCloudPlatform:main Dec 14, 2024
22 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants