Exception when try to save kafka avro data into clickhouse

[licongchao]

i tried to use this connector to save kafka topic data into clickhouse with kafka connector, encountered below exceptions.
Caused by: java.lang.NullPointerException\n\tat com.clickhouse.kafka.connect.sink.db.ClickHouseWriter.validateDataSchema(ClickHouseWriter.java:150)\n\tat com.clickhouse.kafka.connect.sink.db.ClickHouseWriter.doInsertRawBinary(ClickHouseWriter.java:266)\n\tat com.clickhouse.kafka.connect.sink.db.ClickHouseWriter.doInsert(ClickHouseWriter.java:128)\n\tat com.clickhouse.kafka.connect.sink.processing.Processing.doInsert(Processing.java:45)\n\tat com.clickhouse.kafka.connect.sink.processing.Processing.doLogic(Processing.java:102)\n\tat com.clickhouse.kafka.connect.sink.ProxySinkTask.put(ProxySinkTask.java:80)\n\tat com.clickhouse.kafka.connect.sink.ClickHouseSinkTask.put(ClickHouseSinkTask.java:60)\n\tat org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:584)

kafka-avro-console-consumer can consume the data from the topic, and the data shows like
{"entity":"Random1","v":0.5467754725010544}

my kafka connector config is
{ "connector.class": "com.clickhouse.kafka.connect.ClickHouseSinkConnector", "tasks.max": "1", "topics": "simulation", "ssl": "false", "hostname": "clickhouse-db", "database": "sampledatabase", "port": "8123", "exactlyOnce": "false", "username": "default", "value.converter": "io.confluent.connect.avro.AvroConverter", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter.schema.registry.url": "http://schema-registry:8081" }

Did i miss something for the whole solution?

[licongchao]

I looked into the source code, and found the root cause is the type mismatch, Also in com.clickhouse.kafka.connect.sink.db.mapping.Column, it only includes only a few types in clickhouse.
If there's some avro type and clickhouse type mapping document, it will be helpful for the user.

[licongchao]

configuration issue, not a bug, maybe should deliver an enhancement

[mshustov]

configuration issue, not a bug, maybe should deliver an enhancement

@licongchao what would you suggest adding to the documentation to improve the user experience?

[licongchao]

@mshustov Cause Avro has only a few Primitive Types, but there's lots of data types in ClickHouse. If these types are ignored when using, exceptions will occur according to the current implementation.
It has several possible solutions, the first one is to mapping all types within ClickHouse data type. but i think it will cause lots of efforts. The second one is to point out which data types in ClickHouse is already implemented. The root cause of my exception is UINT64 which i'm using but it hasn't been covered, and it cannot pass the data schema validation.
Hope it works, the connector is helpful. thank you.

[mzitnik]

The only types currently not supported by the Sink are bytes & struct as value. We introduce some implicit conversions within Dates see #63 issue [#57]. If you see any other need to implicit conversions, please open an issue, and we will address it.

[Paultagoras]

Given the code added to v0.0.10 by @hamsterready this should be resolved - if it continues to persist, please reopen and thanks!