Block merging of frozen chunks #7665
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #7665 +/- ##
==========================================
+ Coverage 80.06% 81.35% +1.28%
==========================================
Files 190 242 +52
Lines 37181 44890 +7709
Branches 9450 11204 +1754
==========================================
+ Hits 29770 36519 +6749
- Misses 2997 3976 +979
+ Partials 4414 4395 -19 ☔ View full report in Codecov by Sentry. |
gayyappan
reviewed
Feb 7, 2025
gayyappan
approved these changes
Feb 7, 2025
mkindahl
approved these changes
Feb 10, 2025
erimatnor
force-pushed
the
merge-chunks-block-frozen-chunks
branch
2 times, most recently
from
955cc74 to
42de2d4
Compare
It should not be possible to merge a frozen chunk since it is used to tier chunks. If the chunk is merged, it might no longer exist when the tiering happens.
erimatnor
force-pushed
the
merge-chunks-block-frozen-chunks
branch
from
42de2d4 to
dccd04c
Compare
erimatnor
added
force-auto-backport
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Feb 10, 2025
This release contains performance improvements and bug fixes since the 2.18.0 release. We recommend that you upgrade at the next available opportunity. **Features** * timescale#7656 Remove limitation of compression policy for continuous aggregates **Bugfixes** * timescale#6884 Match the Postgres NaN comparison behavior in WHERE clause over compressed tables. * timescale#7600 Fix lock order when dropping index * timescale#7637 Allow EXPLAIN in read-only mode * timescale#7645 Fix DELETE on compressed chunk with non-btree operators * timescale#7649 Allow non-btree operator pushdown in UPDATE/DELETE queries on compressed chunks * timescale#7653 Push down orderby scankeys to Hypercore TAM * timescale#7665 Block merging of frozen chunks * timescale#7673 Don't abort additional INSERTs when hitting first conflict **Thanks** * @bjornuppeke for reporting a problem with INSERT INTO ... ON CONFLICT DO NOTHING on compressed chunks * @ikalafat for reporting a problem with EXPLAIN in read-only mode * @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables. * Timescale community members Jacob and pantonis for reporting issues with slow queries.
Merged
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Feb 10, 2025
This release contains performance improvements and bug fixes since the 2.18.0 release. We recommend that you upgrade at the next available opportunity. **Features** * timescale#7656 Remove limitation of compression policy for continuous aggregates **Bugfixes** * timescale#6884 Match the Postgres NaN comparison behavior in WHERE clause over compressed tables. * timescale#7600 Fix lock order when dropping index * timescale#7637 Allow EXPLAIN in read-only mode * timescale#7645 Fix DELETE on compressed chunk with non-btree operators * timescale#7649 Allow non-btree operator pushdown in UPDATE/DELETE queries on compressed chunks * timescale#7653 Push down orderby scankeys to Hypercore TAM * timescale#7665 Block merging of frozen chunks * timescale#7673 Don't abort additional INSERTs when hitting first conflict **GUCs** * enable_hypercore_scankey_pushdown: Push down qualifiers as scankeys when using Hypercore TAM **Thanks** * @bjornuppeke for reporting a problem with INSERT INTO ... ON CONFLICT DO NOTHING on compressed chunks * @ikalafat for reporting a problem with EXPLAIN in read-only mode * @jakehedlund for reporting the incompatible NaN behavior in WHERE clause over compressed tables. * Timescale community members Jacob and pantonis for reporting issues with slow queries.
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Feb 10, 2025
This release contains performance improvements and bug fixes since the 2.18.0 release. We recommend that you upgrade at the next available opportunity. **Features** * timescale#7656 Remove limitation of compression policy for continuous aggregates **Bugfixes** * timescale#7600 Fix lock order when dropping index * timescale#7637 Allow EXPLAIN in read-only mode * timescale#7645 Fix DELETE on compressed chunk with non-btree operators * timescale#7649 Allow non-btree operator pushdown in UPDATE/DELETE queries on compressed chunks * timescale#7653 Push down orderby scankeys to Hypercore TAM * timescale#7665 Block merging of frozen chunks * timescale#7673 Don't abort additional INSERTs when hitting first conflict **GUCs** * enable_hypercore_scankey_pushdown: Push down qualifiers as scankeys when using Hypercore TAM **Thanks** * @bjornuppeke for reporting a problem with INSERT INTO ... ON CONFLICT DO NOTHING on compressed chunks * @ikalafat for reporting a problem with EXPLAIN in read-only mode * Timescale community members Jacob and pantonis for reporting issues with slow queries.
svenklemm
added a commit
that referenced
this pull request
Feb 10, 2025
This release contains performance improvements and bug fixes since the 2.18.0 release. We recommend that you upgrade at the next available opportunity. **Features** * #7656 Remove limitation of compression policy for continuous aggregates **Bugfixes** * #7600 Fix lock order when dropping index * #7637 Allow EXPLAIN in read-only mode * #7645 Fix DELETE on compressed chunk with non-btree operators * #7649 Allow non-btree operator pushdown in UPDATE/DELETE queries on compressed chunks * #7653 Push down orderby scankeys to Hypercore TAM * #7665 Block merging of frozen chunks * #7673 Don't abort additional INSERTs when hitting first conflict **GUCs** * enable_hypercore_scankey_pushdown: Push down qualifiers as scankeys when using Hypercore TAM **Thanks** * @bjornuppeke for reporting a problem with INSERT INTO ... ON CONFLICT DO NOTHING on compressed chunks * @ikalafat for reporting a problem with EXPLAIN in read-only mode * Timescale community members Jacob and pantonis for reporting issues with slow queries.
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Feb 10, 2025
This release contains performance improvements and bug fixes since the 2.18.0 release. We recommend that you upgrade at the next available opportunity. **Features** * [timescale#7656](timescale#7656) Remove limitation of compression policy for continuous aggregates **Bugfixes** * [timescale#7600](timescale#7600) Fix lock order when dropping index * [timescale#7637](timescale#7637) Allow EXPLAIN in read-only mode * [timescale#7645](timescale#7645) Fix DELETE on compressed chunk with non-btree operators * [timescale#7649](timescale#7649) Allow non-btree operator pushdown in UPDATE/DELETE queries on compressed chunks * [timescale#7653](timescale#7653) Push down orderby scankeys to Hypercore TAM * [timescale#7665](timescale#7665) Block merging of frozen chunks * [timescale#7673](timescale#7673) Don't abort additional INSERTs when hitting first conflict **GUCs** * `enable_hypercore_scankey_pushdown`: Push down qualifiers as scankeys when using Hypercore TAM introduced with [timescale#7653](timescale#7653) **Thanks** * @bjornuppeke for reporting a problem with INSERT INTO ... ON CONFLICT DO NOTHING on compressed chunks * @ikalafat for reporting a problem with EXPLAIN in read-only mode * Timescale community members Jacob and pantonis for reporting issues with slow queries.
Merged
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Feb 10, 2025
This release contains performance improvements and bug fixes since the 2.18.0 release. We recommend that you upgrade at the next available opportunity. **Features** * [timescale#7656](timescale#7656) Remove limitation of compression policy for continuous aggregates **Bugfixes** * [timescale#7600](timescale#7600) Fix lock order when dropping index * [timescale#7637](timescale#7637) Allow EXPLAIN in read-only mode * [timescale#7645](timescale#7645) Fix DELETE on compressed chunk with non-btree operators * [timescale#7649](timescale#7649) Allow non-btree operator pushdown in UPDATE/DELETE queries on compressed chunks * [timescale#7653](timescale#7653) Push down orderby scankeys to Hypercore TAM * [timescale#7665](timescale#7665) Block merging of frozen chunks * [timescale#7673](timescale#7673) Don't abort additional INSERTs when hitting first conflict **GUCs** * `enable_hypercore_scankey_pushdown`: Push down qualifiers as scankeys when using Hypercore TAM introduced with [timescale#7653](timescale#7653) **Thanks** * @bjornuppeke for reporting a problem with INSERT INTO ... ON CONFLICT DO NOTHING on compressed chunks * @ikalafat for reporting a problem with EXPLAIN in read-only mode * Timescale community members Jacob and pantonis for reporting issues with slow queries.
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Feb 10, 2025
This release contains performance improvements and bug fixes since the 2.18.0 release. We recommend that you upgrade at the next available opportunity. **Features** * [timescale#7656](timescale#7656) Remove limitation of compression policy for continuous aggregates **Bugfixes** * [timescale#7600](timescale#7600) Fix lock order when dropping index * [timescale#7637](timescale#7637) Allow EXPLAIN in read-only mode * [timescale#7645](timescale#7645) Fix DELETE on compressed chunk with non-btree operators * [timescale#7649](timescale#7649) Allow non-btree operator pushdown in UPDATE/DELETE queries on compressed chunks * [timescale#7653](timescale#7653) Push down orderby scankeys to Hypercore TAM * [timescale#7665](timescale#7665) Block merging of frozen chunks * [timescale#7673](timescale#7673) Don't abort additional INSERTs when hitting first conflict **GUCs** * `enable_hypercore_scankey_pushdown`: Push down qualifiers as scankeys when using Hypercore TAM introduced with [timescale#7653](timescale#7653) **Thanks** * @bjornuppeke for reporting a problem with INSERT INTO ... ON CONFLICT DO NOTHING on compressed chunks * @ikalafat for reporting a problem with EXPLAIN in read-only mode * Timescale community members Jacob and pantonis for reporting issues with slow queries.
svenklemm
added a commit
to svenklemm/timescaledb
that referenced
this pull request
Feb 10, 2025
This release contains performance improvements and bug fixes since the 2.18.0 release. We recommend that you upgrade at the next available opportunity. **Features** * [timescale#7656](timescale#7656) Remove limitation of compression policy for continuous aggregates **Bugfixes** * [timescale#7600](timescale#7600) Fix lock order when dropping index * [timescale#7637](timescale#7637) Allow EXPLAIN in read-only mode * [timescale#7645](timescale#7645) Fix DELETE on compressed chunk with non-btree operators * [timescale#7649](timescale#7649) Allow non-btree operator pushdown in UPDATE/DELETE queries on compressed chunks * [timescale#7653](timescale#7653) Push down orderby scankeys to Hypercore TAM * [timescale#7665](timescale#7665) Block merging of frozen chunks * [timescale#7673](timescale#7673) Don't abort additional INSERTs when hitting first conflict **GUCs** * `enable_hypercore_scankey_pushdown`: Push down qualifiers as scankeys when using Hypercore TAM introduced with [timescale#7653](timescale#7653) **Thanks** * @bjornuppeke for reporting a problem with INSERT INTO ... ON CONFLICT DO NOTHING on compressed chunks * @ikalafat for reporting a problem with EXPLAIN in read-only mode * Timescale community members Jacob and pantonis for reporting issues with slow queries.
svenklemm
added a commit
that referenced
this pull request
Feb 11, 2025
This release contains performance improvements and bug fixes since the 2.18.0 release. We recommend that you upgrade at the next available opportunity. **Features** * [#7656](#7656) Remove limitation of compression policy for continuous aggregates **Bugfixes** * [#7600](#7600) Fix lock order when dropping index * [#7637](#7637) Allow EXPLAIN in read-only mode * [#7645](#7645) Fix DELETE on compressed chunk with non-btree operators * [#7649](#7649) Allow non-btree operator pushdown in UPDATE/DELETE queries on compressed chunks * [#7653](#7653) Push down orderby scankeys to Hypercore TAM * [#7665](#7665) Block merging of frozen chunks * [#7673](#7673) Don't abort additional INSERTs when hitting first conflict **GUCs** * `enable_hypercore_scankey_pushdown`: Push down qualifiers as scankeys when using Hypercore TAM introduced with [#7653](#7653) **Thanks** * @bjornuppeke for reporting a problem with INSERT INTO ... ON CONFLICT DO NOTHING on compressed chunks * @ikalafat for reporting a problem with EXPLAIN in read-only mode * Timescale community members Jacob and pantonis for reporting issues with slow queries.
philkra
added a commit
that referenced
this pull request
Mar 12, 2025
## 2.19.0 (2025-03-12) This release contains performance improvements and bug fixes since the 2.18.2 release. We recommend that you upgrade at the next available opportunity. **Features** * [#7586](#7586) Vectorized aggregation with grouping by a single text column. * [#7632](#7632) Optimize recompression for chunks without segmentby * [#7655](#7655) Support vectorized aggregation on Hypercore TAM * [#7669](#7669) Add support for merging compressed chunks * [#7701](#7701) Implement a custom compression algorithm for bool columns. It is experimental and can undergo backwards-incompatible changes. For testing, enable it using timescaledb.enable_bool_compression = on. * [#7707](#7707) Support ALTER COLUMN SET NOT NULL on compressed chunks * [#7765](#7765) Allow tsdb as alias for timescaledb in WITH and SET clauses * [#7786](#7786) Show warning for inefficient compress_chunk_time_interval configuration * [#7788](#7788) Add callback to mem_guard for background workers * [#7789](#7789) Do not recompress segmentwise when default order by is empty * [#7790](#7790) Add configurable Incremental CAgg Refresh Policy **Bugfixes** * [#7665](#7665) Block merging of frozen chunks * [#7673](#7673) Don't abort additional INSERTs when hitting first conflict * [#7714](#7714) Fixes a wrong result when compressed NULL values were confused with default values. This happened in very special circumstances with alter table added a new column with a default value, an update and compression in a very particular order. * [#7747](#7747) Block TAM rewrites with incompatible GUC setting * [#7748](#7748) Crash in the segmentwise recompression * [#7764](#7764) Fix compression settings handling in Hypercore TAM * [#7768](#7768) Remove costing index scan of hypertable parent * [#7799](#7799) Handle DEFAULT table access name in ALTER TABLE **Thanks** * @bjornuppeke for reporting a problem with INSERT INTO ... ON CONFLICT DO NOTHING on compressed chunks * @kav23alex for reporting a segmentation fault on ALTER TABLE with DEFAULT Signed-off-by: Philip Krauss <35487337+philkra@users.noreply.github.com>
This was referenced Mar 12, 2025
Closed
Merged
philkra
added a commit
that referenced
this pull request
Mar 18, 2025
## 2.19.0 (2025-03-18) This release contains performance improvements and bug fixes since the 2.18.2 release. We recommend that you upgrade at the next available opportunity. * Improved concurrency of INSERT, UPDATE and DELETE operations on the columnstore by no longer blocking DML statements during the recompression of a chunk. * Improved system performance during Continuous Aggregates refreshes by breaking them into smaller batches which reduces systems pressure and minimizes the risk of spilling to disk. * Faster and more up-to-date results for queries against Continuous Aggregates by materializing the most recent data first (vs old data first in prior versions). * Faster analytical queries with SIMD vectorization of aggregations over text columns and group by over multiple column * Enable optimizing chunk size for faster query performance on the columnstore by adding support for merging columnstore chunks to the merge_chunk API. **Deprecation warning** This is the last minor release supporting PostgreSQL 14. Starting with the minor version of TimescaleDB only Postgres 15, 16 and 17 will be supported. **Downgrading of 2.19.0** This release introduces custom bool compression, if you enable this feature via the `enable_bool_compression` and must downgrade to a previous, please use the [following script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.19.0-downgrade_new_compression_algorithms.sql) to convert the columns back to their previous state. TimescaleDB versions prior to 2.19.0 do not know how to handle this new type. **Features** * [#7586](#7586) Vectorized aggregation with grouping by a single text column. * [#7632](#7632) Optimize recompression for chunks without segmentby * [#7655](#7655) Support vectorized aggregation on Hypercore TAM * [#7669](#7669) Add support for merging compressed chunks * [#7701](#7701) Implement a custom compression algorithm for bool columns. It is experimental and can undergo backwards-incompatible changes. For testing, enable it using timescaledb.enable_bool_compression = on. * [#7707](#7707) Support ALTER COLUMN SET NOT NULL on compressed chunks * [#7765](#7765) Allow tsdb as alias for timescaledb in WITH and SET clauses * [#7786](#7786) Show warning for inefficient compress_chunk_time_interval configuration * [#7788](#7788) Add callback to mem_guard for background workers * [#7789](#7789) Do not recompress segmentwise when default order by is empty * [#7790](#7790) Add configurable Incremental CAgg Refresh Policy **Bugfixes** * [#7665](#7665) Block merging of frozen chunks * [#7673](#7673) Don't abort additional INSERTs when hitting first conflict * [#7714](#7714) Fixes a wrong result when compressed NULL values were confused with default values. This happened in very special circumstances with alter table added a new column with a default value, an update and compression in a very particular order. * [#7747](#7747) Block TAM rewrites with incompatible GUC setting * [#7748](#7748) Crash in the segmentwise recompression * [#7764](#7764) Fix compression settings handling in Hypercore TAM * [#7768](#7768) Remove costing index scan of hypertable parent * [#7799](#7799) Handle DEFAULT table access name in ALTER TABLE **GUCs** * `enable_bool_compression`: enable the BOOL compression algorithm, default: `OFF` * `enable_exclusive_locking_recompression`: enable exclusive locking during recompression (legacy mode), default: `OFF` **Thanks** * @bjornuppeke for reporting a problem with INSERT INTO ... ON CONFLICT DO NOTHING on compressed chunks * @kav23alex for reporting a segmentation fault on ALTER TABLE with DEFAULT --------- Signed-off-by: Philip Krauss <35487337+philkra@users.noreply.github.com> Signed-off-by: Ramon Guiu <ramon@timescale.com> Co-authored-by: Ramon Guiu <ramon@timescale.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
It should not be possible to merge a frozen chunk since it is used to tier chunks. If the chunk is merged, it might no longer exist when the tiering happens.