Skip to content

Implement zstd-based compression #538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 14 commits into
base: master
Choose a base branch
from
+667 −145
Open

Implement zstd-based compression #538

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
a23e084
Implement small test data generator
slabko Oct 30, 2025
f00e81e
Add zstd to the project
slabko Oct 31, 2025
ac982b9
Add zstd compression support to HTTPChunkedStream
slabko Nov 1, 2025
024cefe
Data compression between the CH servers and the drivers (#523)
ilejn Oct 26, 2025
74b6328
Rollback gzip compression, only keep the setting
slabko Nov 2, 2025
3a6b2ba
Pass Accept-Encoding header based on configuration
slabko Nov 2, 2025
44a87ad
Add strict check for mid-stream exceptions
slabko Nov 2, 2025
f7eef61
Update ClickHouse version on CI
slabko Nov 2, 2025
5caec4f
Add extra DSNs to the tests with compression enabled
slabko Nov 3, 2025
865f162
Add documentation to changes in Poco
slabko Nov 3, 2025
753f69f
Add test verifying server respects compression setting
slabko Nov 4, 2025
aaae09c
Cleanup zstd cmake config
slabko Nov 4, 2025
46939e9
Increase buffer sizes to match ZSTD recommendations
slabko Nov 5, 2025
3c6c345
Update poco local changes
slabko Nov 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion .github/workflows/Coverage.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ jobs:
-D CMAKE_POLICY_VERSION_MINIMUM=3.5
-D CMAKE_BUILD_TYPE=Debug
-D ODBC_PROVIDER=UnixODBC
-D TEST_DSN_LIST="ClickHouse DSN (ANSI);ClickHouse DSN (Unicode)"
-D TEST_DSN_LIST="ClickHouse DSN (ANSI);ClickHouse DSN (Unicode);ClickHouse DSN (ANSI) ZSTD;ClickHouse DSN (Unicode) ZSTD"
-D CH_ODBC_ENABLE_CODE_COVERAGE=ON
-D BUILD_TESTING=YES

Expand Down Expand Up @@ -132,6 +132,22 @@ jobs:
Url = http://localhost:8123
DriverLog = yes
DriverLogFile = ${{ github.workspace }}/run/clickhouse-odbc-driver-w.log

[ClickHouse DSN (ANSI) ZSTD]
Driver = ClickHouse ODBC Driver (ANSI)
Description = Test DSN for ClickHouse ODBC Driver (ANSI)
Url = http://${CLICKHOUSE_SERVER_IP}
DriverLog = yes
DriverLogFile = ${{ github.workspace }}/run/clickhouse-odbc-driver.log
Compression = 1

[ClickHouse DSN (Unicode) ZSTD]
Driver = ClickHouse ODBC Driver (Unicode)
Description = Test DSN for ClickHouse ODBC Driver (Unicode)
Url = http://localhost:8123
DriverLog = yes
DriverLogFile = ${{ github.workspace }}/run/clickhouse-odbc-driver-w.log
Compression = 1
EOF

- name: Test - Run C++ unit tests
Expand Down
18 changes: 17 additions & 1 deletion .github/workflows/Linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ jobs:
-DODBC_PROVIDER=${{ matrix.odbc_provider }}
-DCH_ODBC_RUNTIME_LINK_STATIC=${{ fromJSON('{"static-runtime": "ON", "dynamic-runtime": "OFF"}')[matrix.runtime_link] }}
-DCH_ODBC_PREFER_BUNDLED_THIRD_PARTIES=${{ fromJSON('{"bundled-third-parties": "ON", "system-third-parties": "OFF"}')[matrix.third_parties] }}
-DTEST_DSN_LIST="ClickHouse DSN (ANSI);ClickHouse DSN (Unicode)"
-DTEST_DSN_LIST="ClickHouse DSN (ANSI);ClickHouse DSN (Unicode);ClickHouse DSN (ANSI) ZSTD;ClickHouse DSN (Unicode) ZSTD"
-DBUILD_TESTING=YES


Expand Down Expand Up @@ -174,6 +174,22 @@ jobs:
Url = http://localhost:8123
DriverLog = yes
DriverLogFile = ${{ github.workspace }}/run/clickhouse-odbc-driver-w.log

[ClickHouse DSN (ANSI) ZSTD]
Driver = ClickHouse ODBC Driver (ANSI)
Description = Test DSN for ClickHouse ODBC Driver (ANSI)
Url = http://${CLICKHOUSE_SERVER_IP}
DriverLog = yes
DriverLogFile = ${{ github.workspace }}/run/clickhouse-odbc-driver.log
Compression = 1

[ClickHouse DSN (Unicode) ZSTD]
Driver = ClickHouse ODBC Driver (Unicode)
Description = Test DSN for ClickHouse ODBC Driver (Unicode)
Url = http://localhost:8123
DriverLog = yes
DriverLogFile = ${{ github.workspace }}/run/clickhouse-odbc-driver-w.log
Compression = 1
EOF

# Run all tests except those that were run in "Test - unit tests" step.
Expand Down
18 changes: 17 additions & 1 deletion .github/workflows/Sanitizer.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ jobs:
cmake -S ${{ github.workspace }}/source -B ${{ github.workspace }}/build
-DCMAKE_POLICY_VERSION_MINIMUM=3.5
-DCMAKE_BUILD_TYPE=${{ matrix.build_type }}
-DTEST_DSN_LIST="ClickHouse DSN (ANSI);ClickHouse DSN (Unicode)"
-DTEST_DSN_LIST="ClickHouse DSN (ANSI);ClickHouse DSN (Unicode);ClickHouse DSN (ANSI) ZSTD;ClickHouse DSN (Unicode) ZSTD"
-DSANITIZE=${{ matrix.sanitizer }}
-DBUILD_TESTING=YES

Expand Down Expand Up @@ -127,6 +127,22 @@ jobs:
Url = http://localhost:8123
DriverLog = yes
DriverLogFile = ${{ github.workspace }}/run/clickhouse-odbc-driver-w.log

[ClickHouse DSN (ANSI) ZSTD]
Driver = ClickHouse ODBC Driver (ANSI)
Description = Test DSN for ClickHouse ODBC Driver (ANSI)
Url = http://${CLICKHOUSE_SERVER_IP}
DriverLog = yes
DriverLogFile = ${{ github.workspace }}/run/clickhouse-odbc-driver.log
Compression = 1

[ClickHouse DSN (Unicode) ZSTD]
Driver = ClickHouse ODBC Driver (Unicode)
Description = Test DSN for ClickHouse ODBC Driver (Unicode)
Url = http://localhost:8123
DriverLog = yes
DriverLogFile = ${{ github.workspace }}/run/clickhouse-odbc-driver-w.log
Compression = 1
EOF

- name: Test - Run C++ integration tests
Expand Down
16 changes: 15 additions & 1 deletion .github/workflows/Windows.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ jobs:
-DODBC_PROVIDER=${{ matrix.odbc_provider }}
-DCH_ODBC_RUNTIME_LINK_STATIC=OFF
-DCH_ODBC_PREFER_BUNDLED_THIRD_PARTIES=ON
-DTEST_DSN_LIST="ClickHouse DSN ANSI;ClickHouse DSN Unicode"
-DTEST_DSN_LIST="ClickHouse DSN ANSI;ClickHouse DSN Unicode;ClickHouse DSN ANSI ZSTD;ClickHouse DSN Unicode ZSTD"
-DBUILD_TESTING=YES

- name: Build
Expand Down Expand Up @@ -152,6 +152,20 @@ jobs:
-Platform "$platform" `
-SetPropertyValue @("Url=http://127.0.0.1:8123/", "User=default")

Add-OdbcDsn `
-Name "ClickHouse DSN ANSI ZSTD" `
-DriverName "ClickHouse ODBC Driver (ANSI)" `
-DsnType "System" `
-Platform "$platform" `
-SetPropertyValue @("Url=http://127.0.0.1:8123/", "User=default", "Compression=1")

Add-OdbcDsn `
-Name "ClickHouse DSN Unicode ZSTD" `
-DriverName "ClickHouse ODBC Driver (Unicode)" `
-DsnType "System" `
-Platform "$platform" `
-SetPropertyValue @("Url=http://127.0.0.1:8123/", "User=default", "Compression=1")

Write-Host "List of enabled DSNs:"
Get-OdbcDsn -DsnType "System"

Expand Down
18 changes: 17 additions & 1 deletion .github/workflows/macOS.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ jobs:
-DOPENSSL_ROOT_DIR=/usr/local/opt/openssl
-DCH_ODBC_RUNTIME_LINK_STATIC=${{ fromJSON('{"static-runtime": "ON", "dynamic-runtime": "OFF"}')[matrix.runtime_link] }}
-DCH_ODBC_PREFER_BUNDLED_THIRD_PARTIES=${{ fromJSON('{"bundled-third-parties": "ON", "system-third-parties": "OFF"}')[matrix.third_parties] }}
-DTEST_DSN_LIST="ClickHouse DSN (ANSI);ClickHouse DSN (Unicode)"
-DTEST_DSN_LIST="ClickHouse DSN (ANSI);ClickHouse DSN (Unicode);ClickHouse DSN (ANSI) ZSTD;ClickHouse DSN (Unicode) ZSTD"
-DBUILD_TESTING=YES
- name: Build
Expand Down Expand Up @@ -181,6 +181,22 @@ jobs:
Url = http://${CLICKHOUSE_SERVER_IP}
DriverLog = yes
DriverLogFile = ${{ github.workspace }}/run/clickhouse-odbc-driver-w.log
[ClickHouse DSN (ANSI) ZSTD]
Driver = ClickHouse ODBC Driver (ANSI)
Description = Test DSN for ClickHouse ODBC Driver (ANSI)
Url = http://${CLICKHOUSE_SERVER_IP}
DriverLog = yes
DriverLogFile = ${{ github.workspace }}/run/clickhouse-odbc-driver.log
Compression = 1
[ClickHouse DSN (Unicode) ZSTD]
Driver = ClickHouse ODBC Driver (Unicode)
Description = Test DSN for ClickHouse ODBC Driver (Unicode)
Url = http://${CLICKHOUSE_SERVER_IP}
DriverLog = yes
DriverLogFile = ${{ github.workspace }}/run/clickhouse-odbc-driver-w.log
Compression = 1
EOF
# Run all tests except those that were run in "Test - unit tests" step.
Expand Down
3 changes: 3 additions & 0 deletions .gitmodules
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,3 +38,6 @@
[submodule "contrib/asio"]
path = contrib/asio
url = https://github.com/chriskohlhoff/asio
[submodule "contrib/zstd"]
path = contrib/zstd
url = https://github.com/facebook/zstd
5 changes: 3 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,8 +109,9 @@ The list of DSN parameters recognized by the driver is as follows:
| `HugeIntAsString` | `off` | Report integer column types that may underflow or overflow 64-bit signed integer (`SQL_BIGINT`) as a `String`/`SQL_VARCHAR` |
| `DriverLog` | `on` if `CMAKE_BUILD_TYPE` is `Debug`, `off` otherwise | Enable or disable the extended driver logging |
| `DriverLogFile` | `\temp\clickhouse-odbc-driver.log` on Windows, `/tmp/clickhouse-odbc-driver.log` otherwise | Path to the extended driver log file (used when `DriverLog` is `on`) |
| `AutoSessionId` | `off` | Auto generate session_id required to use some features of CH (e.g. TEMPORARY TABLE) |
| `ClientName` | empty | Sets additional information about the calling application. This string will be used as a prefix for the User-Agent header.
| `AutoSessionId` | `off` | Auto generate session_id required to use some features of CH (e.g. TEMPORARY TABLE) |
| `ClientName` | empty | Sets additional information about the calling application. This string will be used as a prefix for the User-Agent header. |
| `Compression` | `off` | Enables compression for data sent from ClickHouse to the driver |


### URL query string
Expand Down
1 change: 1 addition & 0 deletions contrib/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ add_contrib (unixodbc-cmake unixodbc)
add_contrib (nanodbc-cmake nanodbc)
add_contrib (lz4-cmake lz4)
add_contrib (double-conversion-cmake double-conversion)
add_contrib (zstd-cmake zstd)

if (OS_DARWIN OR OS_LINUX)
add_contrib (openssl-cmake openssl)
Expand Down
28 changes: 25 additions & 3 deletions contrib/poco/LocalChanges.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,15 +81,15 @@ disable this, I changed the Windows Secure Channel configuration flags.

--------------------------------------------------------------------------------
commit 1370b0a9f40801801d433aa4bab03f06e5b26710<br/>
Author: Andrew Slabko <andrew@slabko.com><br/>
Author: Andrew Slabko<br/>
Date: Fri Oct 17 18:05:58 2025 +0200<br/>

commit 1ba8988bd54f30d31114e7959c4ac0350511d044<br/>
Author: Andrew Slabko <andrew@slabko.com><br/>
Author: Andrew Slabko<br/>
Date: Fri Oct 17 20:59:04 2025 +0200<br/>

commit a0b391b829f670e06f9b67ce92bab45c75bfd983<br/>
Author: Andrew Slabko <andrew@slabko.com><br/>
Author: Andrew Slabko<br/>
Date: Fri Oct 18 19:58:04 2025 +0200<br/>

Poco::Net::HTTPChunkedStreamBuf was largely rewritten due to incorrect handling
Expand Down Expand Up @@ -139,3 +139,25 @@ Due to these issues, HTTPChunkedStreamBuf was effectively rewritten but remains
in the Poco::Net target. The class is a friend of several others in the library,
and creating a new one would not be practical. The original class was already
broken, so it was modified in place.

--------------------------------------------------------------------------------
commit ac982b98c52f7a65cb699d2017dadb16490df82d<br/>
Author: Andrew Slabko<br/>
Date: Sat Nov 1 19:00:21 2025 +0100<br/>
Comment on lines +145 to +146
Copy link

@windsurf-bot windsurf-bot bot Nov 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The dates in the newly added commit entries (lines 145, 157) show years as 2025, which appears to be incorrect since we're currently in 2023. This could cause confusion for anyone reading the documentation.


Implemented Zstd decompression for input data received from the ClickHouse
server. Most of the work still resides in the already rewritten
HTTPChunkedStreamBuf. However, since the parameter indicating whether
compression should be enabled or not must be read from the response headers and
passed to the HTTPChunkedStreamBuf instance, the HTTPClientSession class was
also slightly modified.

--------------------------------------------------------------------------------
commit 46939e991ce128d8f7e4e9282b9d81ea1e810908<br/>
Author: Andrew Slabko<br/>
Date: Wed Nov 5 10:11:44 2025 +0100<br/>

The amount of data that HTTPChunkStream attempts to read from
HTTPChunkedStreamBuf depends on the fixed size defined by the
HTTPBufferAllocator::BUFFER_SIZE constant. This constant was increased to 128 KB
to match the recommended output buffer size for ZSTD decompression.
2 changes: 1 addition & 1 deletion contrib/poco/Net/CMakeLists.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ add_library(Net ${SRCS})
set_target_properties(Net PROPERTIES CXX_STANDARD 17)
add_library(Poco::Net ALIAS Net)

target_link_libraries(Net PUBLIC Poco::Foundation)
target_link_libraries(Net PUBLIC Poco::Foundation ch_contrib::zstd)
# Windows need additional libraries
if(WIN32)
target_link_libraries(Net PUBLIC "iphlpapi.lib")
Expand Down
2 changes: 1 addition & 1 deletion contrib/poco/Net/include/Poco/Net/HTTPBufferAllocator.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ class Net_API HTTPBufferAllocator

enum
{
BUFFER_SIZE = 4096
BUFFER_SIZE = 1024UL * 128 // Increase buffer size to match zstd's recommended output buffer size
};

private:
Expand Down
Loading
Loading