Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error running DDL Query - when trying to create a new ReplicatedReplacingMergeTree table #503

Closed
Selfeer opened this issue Mar 18, 2024 · 3 comments · Fixed by #535
Closed
Assignees
Labels
high-priority lightweight Issues related to Lightweight version qa-verified label to mark issues that were verified by QA
Milestone

Comments

@Selfeer
Copy link
Collaborator

Selfeer commented Mar 18, 2024

When creating tables with statements from https://github.com/Altinity/clickhouse-sink-connector/blob/develop/sink-connector-lightweight/sql/init_mysql.sql file in MySQL, the tables with the data seem to be correctly replicated in ClickHouse but when trying to create a new table in MySQL after that, for example,

CREATE TABLE test (id INT);

The new table is not replicated in ClickHouse anymore and we get the following error in sink-connector logs:

clickhouse-sink-connector-lt_1  | 70446 2024-03-18 16:35:20.614 [Sink connector Debezium Event Thread] INFO com.altinity.clickhouse.debezium.embedded.cdc.DebeziumChangeEventCapture  - Executed Source DB DDL: CREATE TABLE `test` (
clickhouse-sink-connector-lt_1  |   `my_row_id` bigint unsigned NOT NULL AUTO_INCREMENT /*!80023 INVISIBLE */,
clickhouse-sink-connector-lt_1  |   `id` int DEFAULT NULL,
clickhouse-sink-connector-lt_1  |   PRIMARY KEY (`my_row_id`)
clickhouse-sink-connector-lt_1  | ) Snapshot:false
clickhouse-sink-connector-lt_1  | 70446 2024-03-18 16:35:20.614 [Sink connector Debezium Event Thread] INFO com.altinity.clickhouse.debezium.embedded.cdc.DebeziumChangeEventCapture  - ClickHouse DDL: CREATE TABLE `test`(`my_row_id` Int64 NOT NULL ,`id` Nullable(Int32),`_version` UInt64,`is_deleted` UInt8)Engine=ReplicatedReplacingMergeTree('/clickhouse/tables/{shard}/', '{replica}', _version, is_deleted) ORDER BY (`my_row_id`)
clickhouse-sink-connector-lt_1  | 70455 2024-03-18 16:35:20.623 [Sink connector Debezium Event Thread] ERROR com.altinity.clickhouse.debezium.embedded.cdc.DebeziumChangeEventCapture  - Error running DDL Query: java.sql.SQLException: Code: 47. DB::Exception: Missing columns: 'customerNumber' while processing query: 'customerNumber', required columns: 'customerNumber' 'customerNumber'. (UNKNOWN_IDENTIFIER) (version 23.8.5.16 (official build))
clickhouse-sink-connector-lt_1  | , server ClickHouseNode [uri=http://clickhouse:8123/test, options={custom_settings=allow_experimental_object_type=1,insert_allow_materialized_columns=1,client_name=Client_1}]@174802180

Here are the configurations:

auto.create.tables: "true"
auto.create.tables.replicated: "true"
enable.snapshot.ddl: "true"
snapshot.mode: "initial"

sink-connector build: altinityinfra/clickhouse-sink-connector:497-15f5a0397ecf52f9b5e381eb6097b7da03c34ec7-lt

@Selfeer Selfeer added the lightweight Issues related to Lightweight version label Mar 18, 2024
@Selfeer
Copy link
Collaborator Author

Selfeer commented Mar 18, 2024

Related to #473

@subkanthi subkanthi self-assigned this Mar 18, 2024
@subkanthi
Copy link
Collaborator

subkanthi commented Mar 18, 2024

image

The query that is executed does not have the type column, for some reason the server expects Type column.

73176 [Sink connector Debezium Event Thread] INFO com.altinity.clickhouse.debezium.embedded.cdc.DebeziumChangeEventCapture  - Executed Source DB DDL: create table test_mysql_table (id int, name varchar(255)) Snapshot:false
73176 [Sink connector Debezium Event Thread] INFO  com.altinity.clickhouse.debezium.embedded.cdc.DebeziumChangeEventCapture  - ClickHouse DDL: CREATE TABLE test_mysql_table(id Nullable(Int32),name Nullable(String),`_sign` Int8,`_version` UInt64)Engine=ReplicatedReplacingMergeTree('/clickhouse/tables/{shard}/', '{replica}', _version) ORDER BY tuple()
7

@svb-alt svb-alt added this to the 2.1.0 milestone Apr 4, 2024
@svb-alt svb-alt removed the 2.0.1 label Apr 4, 2024
@subkanthi
Copy link
Collaborator

subkanthi commented Apr 16, 2024

This looks like its related to not adding on CLUSTER '{cluster}' in the table schema.
related to #540

@subkanthi subkanthi linked a pull request Apr 18, 2024 that will close this issue
@Selfeer Selfeer added the qa-verified label to mark issues that were verified by QA label May 15, 2024
@Selfeer Selfeer closed this as completed May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
high-priority lightweight Issues related to Lightweight version qa-verified label to mark issues that were verified by QA
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants