Cassandra DML Generator - Spanner to SourceDB

[taherkl]

Implement CassandraDMLGenerator for Generating Cassandra-Specific DML Statements

Description:
This PR introduces the CassandraDMLGenerator class, which implements the IDMLGenerator interface to handle the generation of Cassandra-specific Data Manipulation Language (DML) statements. The primary features and enhancements included in this implementation are as follows:

Key Features:

• DML Generation Support:
  • Implements the getDMLStatement method to construct Cassandra-compatible DML statements (e.g., INSERT, UPDATE, DELETE, and UPSERT).
  • Validates and maps input data against the schema to ensure correct column names and data types.
• Flexible Data Handling:
  • Supports primary key and non-primary key handling for both insert and update operations.
  • Includes mechanisms for handling complex data types and converting them to Cassandra-compatible formats.
• Timestamp Management:
  • Adds support for appending USING TIMESTAMP in queries, leveraging precise control over data consistency in Cassandra.
• Robust Error Handling:
  • Includes logging for unknown options, invalid data types, and schema mismatches.
  • Throws appropriate exceptions for unsupported configurations and invalid inputs.

[codecov]

Codecov Report

Attention: Patch coverage is 86.91589% with 42 lines in your changes missing coverage. Please review.

Project coverage is 46.83%. Comparing base (8217f1a) to head (b55568e).
Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
...v2/templates/dbutils/dml/CassandraTypeHandler.java	84.39%	16 Missing and 11 partials ⚠️
...2/templates/dbutils/dml/CassandraDMLGenerator.java	90.40%	5 Missing and 7 partials ⚠️
...plates/dbutils/processor/InputRecordProcessor.java	40.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2120      +/-   ##
============================================
+ Coverage     46.72%   46.83%   +0.10%     
- Complexity     3976     4328     +352     
============================================
  Files           872      874       +2     
  Lines         51681    51987     +306     
  Branches       5402     5441      +39     
============================================
+ Hits          24150    24348     +198     
- Misses        25813    25900      +87     
- Partials       1718     1739      +21     

Components	Coverage Δ
spanner-templates	68.76% <86.91%> (-0.05%)	⬇️
spanner-import-export	65.52% <ø> (-0.06%)	⬇️
spanner-live-forward-migration	76.50% <100.00%> (-1.01%)	⬇️
spanner-live-reverse-replication	78.66% <86.91%> (+0.20%)	⬆️
spanner-bulk-migration	87.87% <100.00%> (+<0.01%)	⬆️

Files with missing lines	Coverage Δ
.../teleport/v2/spanner/migrations/schema/Schema.java	90.16% <100.00%> (+0.16%)	⬆️
.../v2/templates/dbutils/dao/source/CassandraDao.java	100.00% <100.00%> (ø)
...ates/dbutils/processor/SourceProcessorFactory.java	91.80% <100.00%> (+1.32%)	⬆️
...leport/v2/templates/transforms/SourceWriterFn.java	85.21% <ø> (ø)
...plates/dbutils/processor/InputRecordProcessor.java	87.71% <40.00%> (-4.74%)	⬇️
...2/templates/dbutils/dml/CassandraDMLGenerator.java	90.40% <90.40%> (ø)
...v2/templates/dbutils/dml/CassandraTypeHandler.java	81.57% <84.39%> (-1.84%)	⬇️

... and 19 files with indirect coverage changes

[VardhanThigle]

Patch coverage is only 33%.
Could we please improve the UT coverage?

[VardhanThigle]

Please consider building value classes over creating mocks for the same.

[pawankashyapollion]

Please consider building value classes over creating mocks for the same.

we are covering this via DML UT there we are creating Value class based on Resources

[pawankashyapollion]

Please consider building value classes over creating mocks for the same.

Address the same

[pawankashyapollion]

Patch coverage is only 33%. Could we please improve the UT coverage?

@VardhanThigle I have not see that we have written any test case of input Record Processor, Do we need to write UT for the existing file as well

[pawankashyapollion]

Please consider building value classes over creating mocks for the same.

Addressed @VardhanThigle

[taherkl]

Patch coverage is only 33%. Could we please improve the UT coverage?

@VardhanThigle I have not see that we have written any test case of input Record Processor, Do we need to write UT for the existing file as well

@pawankashyapollion Vardhan had put up this comment initially when the patch was at 33% before we pushed the extensive UTs and moved up to 84%.

[pawankashyapollion]

before

@taherkl, before we pushed the extensive UTs, we moved up to 84%.

[taherkl]

Hi @VardhanThigle Please review and close the below comments as well.

#2120 (comment)
#2120 (comment)

[pawankashyapollion]

Implement CassandraDMLGenerator for Generating Cassandra-Specific DML Statements

Description: This PR introduces the CassandraDMLGenerator class, which implements the IDMLGenerator interface to handle the generation of Cassandra-specific Data Manipulation Language (DML) statements. The primary features and enhancements included in this implementation are as follows:

Key Features:

• DML Generation Support:

  • Implements the getDMLStatement method to construct Cassandra-compatible DML statements (e.g., INSERT, UPDATE, DELETE, and UPSERT).
  • Validates and maps input data against the schema to ensure correct column names and data types.

• Flexible Data Handling:

  • Supports primary key and non-primary key handling for both insert and update operations.
  • Includes mechanisms for handling complex data types and converting them to Cassandra-compatible formats.

• Timestamp Management:

  • Adds support for appending USING TIMESTAMP in queries, leveraging precise control over data consistency in Cassandra.

• Robust Error Handling:

  • Includes logging for unknown options, invalid data types, and schema mismatches.
  • Throws appropriate exceptions for unsupported configurations and invalid inputs.

@shreyakhajanchi we are validating message generated during Exception
#2120 (comment)

[pawankashyapollion]

Implement CassandraDMLGenerator for Generating Cassandra-Specific DML Statements
Description: This PR introduces the CassandraDMLGenerator class, which implements the IDMLGenerator interface to handle the generation of Cassandra-specific Data Manipulation Language (DML) statements. The primary features and enhancements included in this implementation are as follows:
Key Features:

• DML Generation Support:

  • Implements the getDMLStatement method to construct Cassandra-compatible DML statements (e.g., INSERT, UPDATE, DELETE, and UPSERT).
  • Validates and maps input data against the schema to ensure correct column names and data types.

• Flexible Data Handling:

  • Supports primary key and non-primary key handling for both insert and update operations.
  • Includes mechanisms for handling complex data types and converting them to Cassandra-compatible formats.

• Timestamp Management:

  • Adds support for appending USING TIMESTAMP in queries, leveraging precise control over data consistency in Cassandra.

• Robust Error Handling:

  • Includes logging for unknown options, invalid data types, and schema mismatches.
  • Throws appropriate exceptions for unsupported configurations and invalid inputs.

@shreyakhajanchi we are validating message generated during Exception #2120 (comment)

IllegalArgumentException exception =
assertThrows(
IllegalArgumentException.class,
() -> {
PreparedStatementValueObject result =
getColumnValueByType(spannerColDef, sourceColDef, valuesJson, null);
CassandraTypeHandler.castToExpectedType(result.dataType(), result.value());
});
assertEquals("Error converting value for cassandraType: date", exception.getMessage());

[shreyakhajanchi]

LGTM. Will approve and merge the PR once all the github actions succeed.