NoHttpResponseException while intensive inserts

[sneakythr0ws]

Hi!
I catch an error using sendStream in RowBinary format intensive queries(1M rows in batch every 3 second) One batch inserts in ~10-15 seconds.
Clickhouse machine cpu usage is about 25%.
It appears after 5 minutes of job start.

I tried to catch problematic batch and insert in separately. Everything was fine.

clickouse-jdbc 0.2.4

#ClickHouse/ClickHouse#11766

Sample of usage

try (ClickHouseStatement statement = clickHouseDataSource.getConnection().createStatement()) {
                statement.write().send(QUERY_INSERT, batch, ClickHouseFormat.RowBinary);

Simple schema

CREATE TABLE IF NOT EXISTS t1
(
    timestamp   DateTime,
    sid         FixedString(10),
    url1        String,
    url2        String,
    importMonth UInt32 default toYYYYMM(now()),
    sidHash     UInt32 DEFAULT xxHash32(sid),
    url1Domain  String DEFAULT domainWithoutWWW(url1)
)
    ENGINE = MergeTree
        PARTITION BY (importMonth)
        ORDER BY (sidHash);

Timeouts config

clickHouseDataSource.getProperties().setSessionTimeout(600_000L);
        clickHouseDataSource.getProperties().setConnectionTimeout(600_000);
        clickHouseDataSource.getProperties().setSocketTimeout(600_000);
        clickHouseDataSource.getProperties().setDataTransferTimeout(600_000);
        clickHouseDataSource.getProperties().setKeepAliveTimeout(600_000);

Exception

Caused by: org.apache.http.NoHttpResponseException: server:8123 failed to respond
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
	at ru.yandex.clickhouse.ClickHouseStatementImpl.sendStream(ClickHouseStatementImpl.java:849)
	... 13 common frames omitted

Error in clickhouse log

2020.06.18 15:10:11.079888 [ 966 ] {c2912eff-d759-4b1b-80b3-151e5a96196a} <Error> executeQuery: Code: 23, e.displayText() = DB::Exception: Cannot read from istream at offset 22020096 (version 20.4.4.18 (official build)) (from server:53702) (in query: INSERT INTO tabl1(timestamp,sid,url1,url2) FORMAT RowBinary ), Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x104191d0 in /usr/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x8fff8ad in /usr/bin/clickhouse
2. ? @ 0x90db204 in /usr/bin/clickhouse
3. DB::CompressedReadBufferBase::readCompressedData(unsigned long&, unsigned long&) @ 0xcdf7dce in /usr/bin/clickhouse
4. DB::CompressedReadBuffer::nextImpl() @ 0xcdf616b in /usr/bin/clickhouse
5. DB::ConcatReadBuffer::nextImpl() @ 0x90e8912 in /usr/bin/clickhouse
6. DB::ConcatReadBuffer::nextImpl() @ 0x90e8912 in /usr/bin/clickhouse
7. DB::ReadBuffer::readStrict(char*, unsigned long) @ 0x903b525 in /usr/bin/clickhouse
8. DB::DataTypeString::deserializeBinary(DB::IColumn&, DB::ReadBuffer&) const @ 0xcf3ade5 in /usr/bin/clickhouse
9. DB::BinaryRowInputFormat::readRow(std::__1::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, DB::RowReadExtension&) @ 0xdb683af in /usr/bin/clickhouse
10. DB::IRowInputFormat::generate() @ 0xdb5d151 in /usr/bin/clickhouse
11. DB::ISource::work() @ 0xdaf3d8b in /usr/bin/clickhouse
12. DB::InputStreamFromInputFormat::readImpl() @ 0xdab99fd in /usr/bin/clickhouse
13. DB::IBlockInputStream::read() @ 0xce4825d in /usr/bin/clickhouse
14. DB::AddingDefaultsBlockInputStream::readImpl() @ 0xce3896b in /usr/bin/clickhouse
15. DB::IBlockInputStream::read() @ 0xce4825d in /usr/bin/clickhouse
16. DB::InputStreamFromASTInsertQuery::readImpl() @ 0xd1cbc39 in /usr/bin/clickhouse
17. DB::IBlockInputStream::read() @ 0xce4825d in /usr/bin/clickhouse
18. DB::copyData(DB::IBlockInputStream&, DB::IBlockOutputStream&, std::__1::atomic<bool>*) @ 0xce7717e in /usr/bin/clickhouse
19. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>) @ 0xd54d5ab in /usr/bin/clickhouse
20. DB::HTTPHandler::processQuery(Poco::Net::HTTPServerRequest&, HTMLForm&, Poco::Net::HTTPServerResponse&, DB::HTTPHandler::Output&) @ 0x90e47fc in /usr/bin/clickhouse
21. DB::HTTPHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&) @ 0x90e8256 in /usr/bin/clickhouse
22. Poco::Net::HTTPServerConnection::run() @ 0x102c9b83 in /usr/bin/clickhouse
23. Poco::Net::TCPServerConnection::start() @ 0x10304f4b in /usr/bin/clickhouse
24. Poco::Net::TCPServerDispatcher::run() @ 0x103053db in /usr/bin/clickhouse
25. Poco::PooledThread::run() @ 0x104b2fa6 in /usr/bin/clickhouse
26. Poco::ThreadImpl::runnableEntry(void*) @ 0x104ae260 in /usr/bin/clickhouse
27. start_thread @ 0x7fa3 in /lib/x86_64-linux-gnu/libpthread-2.28.so
28. __clone @ 0xf94cf in /lib/x86_64-linux-gnu/libc-2.28.so

2020.06.18 15:10:11.080144 [ 966 ] {} <Error> DynamicQueryHandler: Code: 23, e.displayText() = DB::Exception: Cannot read from istream at offset 22020096, Stack trace (when copying this message, always include the lines below):

0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x104191d0 in /usr/bin/clickhouse
1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x8fff8ad in /usr/bin/clickhouse
2. ? @ 0x90db204 in /usr/bin/clickhouse
3. DB::CompressedReadBufferBase::readCompressedData(unsigned long&, unsigned long&) @ 0xcdf7dce in /usr/bin/clickhouse
4. DB::CompressedReadBuffer::nextImpl() @ 0xcdf616b in /usr/bin/clickhouse
5. DB::ConcatReadBuffer::nextImpl() @ 0x90e8912 in /usr/bin/clickhouse
6. DB::ConcatReadBuffer::nextImpl() @ 0x90e8912 in /usr/bin/clickhouse
7. DB::ReadBuffer::readStrict(char*, unsigned long) @ 0x903b525 in /usr/bin/clickhouse
8. DB::DataTypeString::deserializeBinary(DB::IColumn&, DB::ReadBuffer&) const @ 0xcf3ade5 in /usr/bin/clickhouse
9. DB::BinaryRowInputFormat::readRow(std::__1::vector<COW<DB::IColumn>::mutable_ptr<DB::IColumn>, std::__1::allocator<COW<DB::IColumn>::mutable_ptr<DB::IColumn> > >&, DB::RowReadExtension&) @ 0xdb683af in /usr/bin/clickhouse
10. DB::IRowInputFormat::generate() @ 0xdb5d151 in /usr/bin/clickhouse
11. DB::ISource::work() @ 0xdaf3d8b in /usr/bin/clickhouse
12. DB::InputStreamFromInputFormat::readImpl() @ 0xdab99fd in /usr/bin/clickhouse
13. DB::IBlockInputStream::read() @ 0xce4825d in /usr/bin/clickhouse
14. DB::AddingDefaultsBlockInputStream::readImpl() @ 0xce3896b in /usr/bin/clickhouse
15. DB::IBlockInputStream::read() @ 0xce4825d in /usr/bin/clickhouse
16. DB::InputStreamFromASTInsertQuery::readImpl() @ 0xd1cbc39 in /usr/bin/clickhouse
17. DB::IBlockInputStream::read() @ 0xce4825d in /usr/bin/clickhouse
18. DB::copyData(DB::IBlockInputStream&, DB::IBlockOutputStream&, std::__1::atomic<bool>*) @ 0xce7717e in /usr/bin/clickhouse
19. DB::executeQuery(DB::ReadBuffer&, DB::WriteBuffer&, bool, DB::Context&, std::__1::function<void (std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)>) @ 0xd54d5ab in /usr/bin/clickhouse
20. DB::HTTPHandler::processQuery(Poco::Net::HTTPServerRequest&, HTMLForm&, Poco::Net::HTTPServerResponse&, DB::HTTPHandler::Output&) @ 0x90e47fc in /usr/bin/clickhouse
21. DB::HTTPHandler::handleRequest(Poco::Net::HTTPServerRequest&, Poco::Net::HTTPServerResponse&) @ 0x90e8256 in /usr/bin/clickhouse
22. Poco::Net::HTTPServerConnection::run() @ 0x102c9b83 in /usr/bin/clickhouse
23. Poco::Net::TCPServerConnection::start() @ 0x10304f4b in /usr/bin/clickhouse
24. Poco::Net::TCPServerDispatcher::run() @ 0x103053db in /usr/bin/clickhouse
25. Poco::PooledThread::run() @ 0x104b2fa6 in /usr/bin/clickhouse
26. Poco::ThreadImpl::runnableEntry(void*) @ 0x104ae260 in /usr/bin/clickhouse
27. start_thread @ 0x7fa3 in /lib/x86_64-linux-gnu/libpthread-2.28.so
28. __clone @ 0xf94cf in /lib/x86_64-linux-gnu/libc-2.28.so
 (version 20.4.4.18 (official build))

[qoega]

Actually I can reproduce it even in clickhouse-jdbc tests..

[ERROR] testTSV(ru.yandex.clickhouse.integration.WriterTest)  Time elapsed: 3.502 s  <<< FAILURE!
ru.yandex.clickhouse.except.ClickHouseUnknownException: ClickHouse exception, code: 1002, host: localhost, port: 8123; localhost:8123 failed to respond
	at ru.yandex.clickhouse.integration.WriterTest.testTSV(WriterTest.java:70)
Caused by: org.apache.http.NoHttpResponseException: localhost:8123 failed to respond

[den-crane]

it's a duplicate of #290