Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for calculated columns #429

Closed
aadant opened this issue Jan 7, 2024 · 2 comments · Fixed by #433
Closed

Support for calculated columns #429

aadant opened this issue Jan 7, 2024 · 2 comments · Fixed by #433
Assignees
@aadant @subkanthi
Labels
GA-1 All the issues that are issues in release(Scheduled Dec 2023) qa-verified label to mark issues that were verified by QA

Comments

@aadant
Copy link
Collaborator

aadant commented Jan 7, 2024

MySQL table schema can have generated columns which are not understood by CH.

Let us fix this in 2 steps :

  1. the DDL translation should convert generated columns into ALIAS columns
  • clickhouse_loader.py
  • sink-connector schema java converter
  1. if the calculated expression is not understood by CH, it should be overridable (see schema overrides)

Example column :

`concat_key` varchar(1000) COLLATE latin1_general_cs GENERATED ALWAYS AS (concat(one, '-', two) VIRTUAL,

https://dev.mysql.com/doc/refman/8.0/en/create-table-generated-columns.html

MySQL / Postgres should be supported.
@aadant
Copy link
Collaborator Author

aadant commented Jan 7, 2024

python utilities now support generated columns see https://github.com/Altinity/clickhouse-sink-connector/pull/426/files

@subkanthi subkanthi added the GA-1 All the issues that are issues in release(Scheduled Dec 2023) label Jan 8, 2024
@subkanthi subkanthi linked a pull request Jan 10, 2024 that will close this issue
429 support for calculated columns(mysql generate) #433
Merged
@subkanthi subkanthi mentioned this issue Jan 10, 2024
Merged
@Selfeer Selfeer mentioned this issue Feb 8, 2024
Closed
@Selfeer
Copy link
Collaborator

Selfeer commented Feb 15, 2024

The issue was verified by the Altinity QA team and marked as qa-verified.

Build used for testing: altinityinfra/clickhouse-sink-connector:443-678d8ae2567c2c1d89f5430fa57ede619d6ea851-lt

Automated TestFlows tests were added to the develop branch:
https://github.com/Altinity/clickhouse-sink-connector/blob/develop/sink-connector-lightweight/tests/integration/tests/calculated_columns.py

Requirements for the ClickHouse Sink Connector Lightweight were updated.

@Selfeer Selfeer added the qa-verified label to mark issues that were verified by QA label Feb 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aadant aadant

@subkanthi subkanthi

Labels
GA-1 All the issues that are issues in release(Scheduled Dec 2023) qa-verified label to mark issues that were verified by QA
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

429 support for calculated columns(mysql generate) Altinity/clickhouse-sink-connector
3 participants
@aadant @subkanthi @Selfeer