Sink crashes when inserting required LowCardinality columns

[abdelhakimbendjabeur]

Describe the bug

When a table has a required LowCardinality column, the sink fails because of some byte mismatch (see below for log message).
It works fine if the column is nullable.

I have started working on it and managed to reproduce the error using a unit test.

If anybody has an idea on how to quickly fix, it'd be great. In the meantime, I'll see what I can do

Steps to reproduce

Check the Unit Test in this PR #104

Expected behaviour

Data should be inserted correctly

Error log

Caused by: java.util.concurrent.ExecutionException: com.clickhouse.client.ClickHouseException: Code: 33. DB::Exception: Cannot read all data. Bytes read: 66. Bytes expected: 98.: (at row 7)

👇 This is the line that throws the exception

ClickHouseResponse response = future.get()

Configuration

Environment

• Kafka-Connect version:
• Kafka Connect configuration:
• Kafka version:
• Kafka environment:
• OS:

ClickHouse server

• ClickHouse Server version:
• ClickHouse Server non-default settings, if any:
• CREATE TABLE statements for tables involved:
• Sample data for all these tables, use clickhouse-obfuscator if necessary

[nicolae-gorgias]

@abdelhakimbendjabeur It's an interesting issue, LowCardinality shouldn't affect how the data is sent or received, it's just a way to store it.

I was thinking that it was some incompatibility of the click house client and upgraded it to the latest version but didn't help. Also if you will have just 2 not-null LowCardinality fields it doesn't raise an error but it doesn't insert them :D weird.

[mzitnik]

@abdelhakimbendjabeur did you encounter the issue in JSON or Avro format?

[abdelhakimbendjabeur]

@mzitnik I encountered this one in Avro format

[abdelhakimbendjabeur]

We fixed it in #104