[ISSUE #4331]Add top-level interfaces of jdbc source connector

[mxsm]

Fixes #4331

Motivation

Explain the content here.
Explain why you want to make the changes and what problem you're trying to solve.

Modifications

Describe the modifications you've done.

• Add top-level interfaces of jdbc source connector

Documentation

• Does this pull request introduce a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
• If a feature is not applicable for documentation, explain why?
• If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

[codecov]

Codecov Report

Merging #4332 (8a7d799) into master (1e59cdf) will decrease coverage by 0.32%.
The diff coverage is 8.30%.

❗ Current head 8a7d799 differs from pull request most recent head 802b96c. Consider uploading reports for the commit 802b96c to get more accurate results

@@             Coverage Diff              @@
##             master    #4332      +/-   ##
============================================
- Coverage     17.80%   17.49%   -0.32%     
- Complexity     1516     1544      +28     
============================================
  Files           611      663      +52     
  Lines         25598    26441     +843     
  Branches       2377     2439      +62     
============================================
+ Hits           4559     4627      +68     
- Misses        20601    21364     +763     
- Partials        438      450      +12     

Files Changed	Coverage Δ
...pache/eventmesh/connector/jdbc/CatalogChanges.java	0.00% <0.00%> (ø)
...ava/org/apache/eventmesh/connector/jdbc/Field.java	0.00% <0.00%> (ø)
...ache/eventmesh/connector/jdbc/JdbcConnectData.java	0.00% <0.00%> (ø)
...e/eventmesh/connector/jdbc/JdbcDriverMetaData.java	0.00% <0.00%> (ø)
...a/org/apache/eventmesh/connector/jdbc/Payload.java	0.00% <0.00%> (ø)
...va/org/apache/eventmesh/connector/jdbc/Schema.java	0.00% <0.00%> (ø)
...he/eventmesh/connector/jdbc/config/JdbcConfig.java	0.00% <0.00%> (ø)
...ntmesh/connector/jdbc/config/JdbcServerConfig.java	0.00% <0.00%> (ø)
...mesh/connector/jdbc/connection/JdbcConnection.java	0.00% <0.00%> (ø)
...h/connector/jdbc/connection/PreparedParameter.java	0.00% <0.00%> (ø)
... and 43 more

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[Alonexc]

Merging the master branch will fix this build error.

[mxsm]

Merging the master branch will fix this build error.

I will do it late

[qqeasonchen]

@mxsm too many files changed, could you please split it？

[mxsm]

@mxsm too many files changed, could you please split it？

I will

[HattoriHenzo]

@mxsm The way I was looking this implementation is a bit different. What do you think about a wrapper API around vanilla jdbc with the ability to load a database driver depending on the source(MySQL, PostgreSQL, SQL Server etc...) or the sink(MySQL, PostgreSQL, SQL Server etc..).

[mxsm]

@mxsm The way I was looking this implementation is a bit different. What do you think about a wrapper API around vanilla jdbc with the ability to load a database driver depending on the source(MySQL, PostgreSQL, SQL Server etc...) or the sink(MySQL, PostgreSQL, SQL Server etc..).

@Pil0tXia @HattoriHenzo
Thank you for your suggestion. Here is an abstract architecture diagram of my code:
The diagram is divided into three parts:

The main idea of the design is to parse the data model of the database supporting Jdbc into the corresponding EventMesh event model for processing.Provide a universal interface for interactivity with the database layer and load different database drivers through protocols. Here, I provide dynamic loading of database drivers through JdbcConnection. Later, additional database support can be added by simply including the corresponding database JDBC driver. When the module starts, it automatically loads the corresponding driver based on the configured JDBC protocol.

1. The JdbcConnection API wrapper part is mainly responsible for loading different drivers based on the configured JDBC protocol and creating a connection for the engine to use, for example, configuring jdbc:mysql://localhost:3306 would load the MySQL driver.The JdbcConnection can load different database drivers based on different protocols.

2. The abstract engine module is divided into two types: a snapshot engine for processing data snapshots and a CDC engine for processing database CDC. The engine is responsible for converting data into custom events within EventMesh.

3. Task management mainly manages tasks for processing events. Tasks are divided into two categories: tasks for processing snapshot events (these tasks can be closed after snapshot processing is complete), and tasks for processing CDC events.

I will improve the code and ask for your help to review it later.

[HattoriHenzo]

@mxsm thanks for the details. Here is a more detailed schema that represent my approach.

1. The configuration file contains the jdbc driver to load with a factory class
2. The source/sink are implemented using the eventmesh-openconnect-java such as eventmesh-connector-mongodb or eventmesh-connector-redis
3. java.sql classes a wrapped by the source/sink classes to interact with the underlined database

[mxsm]

@HattoriHenzo Thanks for the visual illustration!

1. JdbcConnection provides a wrapper for java.sql, such as obtaining a connection, setting the connection's autocommit property, and more. For specific database queries, JdbcConnection can be subclassed for implementation. JdbcConnection provides general interfaces for things like executing SQL statements.
2. The connection factory is responsible for creating JdbcConnection connections. Here, java.sql.DriverManager is used to automatically load database drivers based on the jdbc protocol configured in a configuration file.
3. For the Jdbc Source, since it needs to support Cdc, the implementation of Cdc is different for different data types. The relevant configuration for Cdc for a specific database is added in the Source configuration file.
4. The JDBC Sink is mainly responsible for inserting data into the corresponding database. Only the general configuration for JDBC needs to be configured, and then the data from the Source is converted into the data needed by the Sink based on the type of database and saved.

The overall architecture design is basically the same as yours, but there are some differences in nomenclature and implementation details. I will split the PR into smaller chunks to facilitate code review by others.

[HattoriHenzo]

@mxsm Thanks for your reply. Your explanation is very clear.

[qqeasonchen]

@mxsm Have you done your split?

[mxsm]

@qqeasonchen
I am still working on further optimizing the overall code and data format sent to EventMesh. The corresponding PR will be submitted in the next few days. The PR will be submitted in the following sections:

1. Top-level interfaces
2. MySQL snapshot processing implementation
3. MySQL CDC processing implementation

After the overall MySQLConnector source is implemented, MySQL, PG, and Oracle sinks will be implemented first.

[mxsm]

@mxsm Have you done your split?

@qqeasonchen Have done

[qqeasonchen]

@mxsm Have you done your split?

@qqeasonchen Have done

137 file changed is too risky.

[mxsm]

@mxsm Have you done your split?

@qqeasonchen Have done

137 file changed is too risky.

@qqeasonchen I will remove the database abstraction module and event module, leaving only the code at the interface level.

[qqeasonchen]

@mxsm Have you done your split?

@qqeasonchen Have done

137 file changed is too risky.

@qqeasonchen I will remove the database abstraction module and event module, leaving only the code at the interface level.

you can submit only jdbc-connector module in one pr, and other changes in other prs

[mxsm]

@qqeasonchen Based on the current PR, the changes are mainly to the jdbc connector module. During the previous implementation, the changes in PR #4131 were included, but PR #4131 has been in the code review and merge state. Now that I have made changes, only the interface-level code is included in the current PR. I was wondering if you think it’s okay to review and merge PR #4131.

[mxsm]

@qqeasonchen The current code only includes the jdbcconnector module, but I need help reviewing the code in PR #4131 to solve a problem in the subsequent SPI loading.

[qqeasonchen]

@qqeasonchen The current code only includes the jdbcconnector module, but I need help reviewing the code in PR #4131 to solve a problem in the subsequent SPI loading.

then we need to review #4131 first, you can ping anyone you want