Random ClickHouse exception, code: 1002 #478
Description
Hello,
I'm facing an UnknownException using the JDBC for ClickHouse
The problem
I got the exception randomly on some queries (any kind). I can execute those queries by hand or re-run the program and they will work. It happen maybe 1 time for 100 000 queries. It's not a big issue, but we are still loosing some data when it happens on an insert.
What is confusing is the absence of log server-side.
How to reproduce
I'm not able to reproduce it, it just happen randomly on different queries at different moment.
Error messages
On client side, I got this exception :
Caused by: java.sql.SQLException: ClickHouse exception, code: 1002, host: 10.4.239.59, port: 8123;
at com.livingobjects.swalo.tools.sql.SqlPool$4.run(SqlPool.java:568) ~[collector-core-3.18.2.jar:na]
... 3 common frames omitted
Caused by: java.sql.SQLException: INSERT INTO `*****`.`*****` ....... (cutted)
... 4 common frames omitted
Caused by: ru.yandex.clickhouse.except.ClickHouseUnknownException: ClickHouse exception, code: 1002, host: 10.4.239.59, port: 8123;
at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.getException(ClickHouseExceptionSpecifier.java:91) ~[clickhouse-jdbc-0.2.3.jar:na]
at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:55) ~[clickhouse-jdbc-0.2.3.jar:na]
at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:28) ~[clickhouse-jdbc-0.2.3.jar:na]
at ru.yandex.clickhouse.ClickHouseStatementImpl.checkForErrorAndThrow(ClickHouseStatementImpl.java:875) ~[clickhouse-jdbc-0.2.3.jar:na]
at ru.yandex.clickhouse.ClickHouseStatementImpl.sendStream(ClickHouseStatementImpl.java:851) ~[clickhouse-jdbc-0.2.3.jar:na]
at ru.yandex.clickhouse.ClickHouseStatementImpl.sendStream(ClickHouseStatementImpl.java:824) ~[clickhouse-jdbc-0.2.3.jar:na]
at ru.yandex.clickhouse.ClickHouseStatementImpl.sendStream(ClickHouseStatementImpl.java:817) ~[clickhouse-jdbc-0.2.3.jar:na]
at ru.yandex.clickhouse.ClickHousePreparedStatementImpl.executeBatch(ClickHousePreparedStatementImpl.java:335) ~[clickhouse-jdbc-0.2.3.jar:na]
at ru.yandex.clickhouse.ClickHousePreparedStatementImpl.executeBatch(ClickHousePreparedStatementImpl.java:320) ~[clickhouse-jdbc-0.2.3.jar:na]
at com.zaxxer.hikari.proxy.StatementProxy.executeBatch(StatementProxy.java:116) ~[HikariCP-java6-2.3.8.jar:na]
at com.zaxxer.hikari.proxy.PreparedStatementJavassistProxy.executeBatch(PreparedStatementJavassistProxy.java) ~[HikariCP-java6-2.3.8.jar:na]
at com.livingobjects.swalo.tools.sql.SqlPool$4.run(SqlPool.java:550) ~[collector-core-3.18.2.jar:na]
... 3 common frames omitted
Caused by: java.lang.Throwable:
at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:53) ~[clickhouse-jdbc-0.2.3.jar:na]
... 13 common frames omittedI also know the precise hour 2020-07-23T10:00:03,891
Knowing this I can check server-side if there is a log for this hour :
- clickhouse-server.err.log :
2020.07.23 09:57:43.960240 [ 15728 ] {99f1324f-4bc3-42e6-a4b6-4e98b8578411} <Warning> ConnectionPoolWithFailover: Connection failed at try №1, reason: Code: 32, e.displayText() = DB::Exception: Attempt to read $
EOF
- clickhouse-server.log :
2020.07.23 10:00:00.488332 [ 15632 ] {} <Debug> MemoryTracker: Peak memory usage: 275.98 MiB.
2020.07.23 10:00:05.966916 [ 15650 ] {} <Trace> SystemLog (system.trace_log): Flushing system log
Additional context
The connection to the server is multi threaded by HikariCP 2.8.3
We are using ClickHouse server version 20.4.5 revision 54434
and JDBC v0.2.3