Skip to content

Integrate latest changes in CS into stable branch #9271

Closed
@zverevgeny

Description

@zverevgeny

notation:
(+!) - already integrated, conflict when cherry-pick again
(+!-) - already integrated, when cherry-pick again tries to rollback previous changes
good - cherry-picked with no conflict
(+0) - already integrated, no changes when cherry-pick again
ignore - do not cherry-pick this time
conflict - there was some conflicts

e3c9723 zverevgeny Sun Sep 15 09:20:22 Turn on GCCountersNormalizer (#8166)
revert ccf98e2

good 991079d zverevgeny Mon Jul 22 09:14:55 delete unused ISSTransactionOperator (#6908)
(+!) 51ddf40 ivanmorozov333 Mon Jul 22 11:46:09 unify indexation (for committed data) and compaction (#6909)
(+!) 0662944 ivanmorozov333 Mon Jul 22 16:50:51 separate merger logic for make new one (#6931)
good 298b189 Semyon Mon Jul 22 21:57:11 Fix flaky KqpOlapScheme.DropTable (#6795)
(+0) 2ca7153 zverevgeny Tue Jul 23 08:15:22 Extend DisabledOnSchemeShard FF for column tables on dedicated bases (#6498)
good 669605c zverevgeny Wed Jul 24 09:30:10 increase CS read timeout (#6963)
(+0) edea306 zverevgeny Wed Jul 24 18:30:33 fix unused (#7046)
good 4598038 ivanmorozov333 Fri Jul 26 10:16:10 add stats test for internal column table from tables store (#6684)
(+0) 9785f6a ivanmorozov333 Fri Jul 26 19:06:17 dont create delete flag column in indexation (#7082)
(+0) 0671f74 zverevgeny Sat Jul 27 11:22:09 Disable modification operations in CS by default (#7129)
conflict ae7146c ivanmorozov333 Sun Jul 28 08:32:13 implement sparse data accessor (#7055)

good 46db233 ivanmorozov333 Mon Jul 29 07:24:00 additional test and addressing fixes (#7168)
(+0) 123a335 ivanmorozov333 Mon Jul 29 14:33:37 fix compaction intervals construction (#7176)
good 13b5991 ivanmorozov333 Mon Jul 29 16:40:28 tx operations start unification (#7178)
good 33bdce5 ivanmorozov333 Mon Jul 29 17:48:17 asymmetric access for sorted key on find position (#7177)
good 75ab2b2 ivanmorozov333 Tue Jul 30 07:01:57 improve performace for stat requests (sysview) (#7211)
good 49a8f16 ivanmorozov333 Tue Jul 30 13:58:38 speed up merging with correct pointers operation (#7230)
good a87900c ivanmorozov333 Tue Jul 30 13:58:54 speed up tables manager in schemeshard (#7229)
(+!-) ee419db ivanmorozov333 Tue Jul 30 17:03:46 fix segfault in case normalization different versions (#7239)
(+0) 74bba08 ivanmorozov333 Wed Jul 31 11:52:15 fix leak on release internal local tx operation (#7280)
good 407a550 ivanmorozov333 Wed Jul 31 16:40:34 fix pk checker (#7266)
good 88d40e9 ivanmorozov333 Wed Jul 31 17:10:46 correct not null flags usage in tests (#7293)
good 40be822 ivanmorozov333 Wed Jul 31 21:16:06 signals for tablet initialization (#7281)

fix_unused: see #7330 and run tests

85d4495 Ivan Jul 15 15:29:47 [KIKIMR-21472] Output the final AST in explain query (#6487)
good ff5d894 ivanmorozov333 Wed Jul 31 21:28:48 switch on aggregations pushdown into CS (#7305)

(+!-) ae8c7f9 ivanmorozov333 Thu Aug 1 06:51:24 fix empty column usage (#7334)
(conflict) b1c5908 ivanmorozov333 Thu Aug 1 11:55:47 fix columns normalization for absent (#7340)
good 6b1225c zverevgeny Thu Aug 1 15:53:05 Move new flag to env from config for compatibility with musty arc recipy (#7272)
good c0c6ab2 Semyon Thu Aug 1 23:42:52 add missing table stats for column shards (#7140)
good d34dcf3 zverevgeny Fri Aug 2 11:34:44 fix clickbench with CS (#7393)
good d4149ad zverevgeny Sun Aug 4 10:00:20 move test TShardReader to a common place (#7441)
good 899ccd7 ivanmorozov333 Sun Aug 4 13:24:09 Overlimit read memory (#7408)
good f319dde ivanmorozov333 Sun Aug 4 13:56:43 make local internal CS persistent snapshot pinger (#7431)
good a1f1ecc ivanmorozov333 Mon Aug 5 11:47:37 Blob memory interval limit (#7444)
conflict 7d65202 Artem Alekseev Mon Aug 5 16:22:49 Fix KqpOlapAggregations.Aggregation_ResultCountAll_Filter (#7423)
good 58580f8 Semyon Tue Aug 6 14:53:20 add tx progress counters (#7407)
good 7ad9e1e Semyon Wed Aug 7 17:07:35 add missing monitoring metrics for column shards (#7314)
good 9dbab08 ivanmorozov333 Thu Aug 8 11:24:19 fix segfault in deserializer stream usage (#7542)
!!!check spaces good 78d1439 Artem Alekseev Thu Aug 8 18:09:29 Add simple stability test for olap queries (#7543)
(+!-) b16060e zverevgeny Fri Aug 9 08:48:32 Delete empty portions normalizer (#7596)
good c59279a Artem Alekseev Fri Aug 9 15:51:38 Fix integer overflow in TTimeSliceLogic::DoCalcWeight (#7606)
conflict a2e8760 ivanmorozov333 Mon Aug 12 09:44:16 change memory planner for CS scan (#7372)
good ef2fc0f zverevgeny Mon Aug 12 19:26:23 delete non-existing ut from muted (#7672)
good 3734eda ivanmorozov333 Tue Aug 13 19:31:50 different request variants need different memory profiles (#7664)
(+!-) 555ebf3 ivanmorozov333 Tue Aug 13 19:32:43 improve chunks splitter performance (#7665)
ignore 63c9b91 ildar-khisambeev Wed Aug 14 16:43:49 add status processing (#6609)
good cb4e391 zverevgeny Wed Aug 14 22:44:30 unmute incorrectly muted olap tests (#7680)
(+0) 009d3c9 ivanmorozov333 Thu Aug 15 12:31:42 scan error processing on restore data (#7813)
good 3d60e1e ivanmorozov333 Thu Aug 15 15:32:47 fix snapshot tracing signals (#7821)
good 752095e ivanmorozov333 Thu Aug 15 16:11:33 fix snapshot control in case sys view requests (#7824)
(+!-) 1159d25 ivanmorozov333 Thu Aug 15 18:06:20 Optional columns feature flags (#7814)
good 83b9579 ivanmorozov333 Fri Aug 16 07:08:28 dont use cpu for not abortable chunks (#7865)
(+0) 8796c8a ivanmorozov333 Fri Aug 16 11:09:10 fix portion kind modification (#7868)
good ac738f9 Semyon Fri Aug 16 14:44:00 Show RuntimeFeatures on portion_stats sys-view (#7673)
(+0) 04beb79 ivanmorozov333 Fri Aug 16 15:36:39 replace as default write modification type (#7894)
good f6476d4 ivanmorozov333 Fri Aug 16 16:08:29 speed up compilation for request processing (#7900)
good c68f701 Artem Alekseev Fri Aug 16 16:34:33 Fix kikimr_runner for ubuntu 20.04 (#7895)
(+0) af37b14 ivanmorozov333 Fri Aug 16 16:56:43 additional tx signals (#7876)
(+0) 7b528af ivanmorozov333 Fri Aug 16 22:21:46 Fix chunks scanner (#7913)
(+0) c03e8b2 ivanmorozov333 Sat Aug 17 07:50:57 speed up filters construction (#7934)
(+0) b524a82 ivanmorozov333 Sun Aug 18 09:29:45 timeout for long_tx operations (#7961)
good 5522a96 ivanmorozov333 Mon Aug 19 07:52:55 fix race on scan start with indexation cleaning blobs (#7968)
good 4226868 ivanmorozov333 Mon Aug 19 13:05:19 refactoring for ranges control (#7758)
good a61fa8d Semyon Tue Aug 20 17:10:31 Add a scenario test for altering tiering config (#7636)
<-- cs progress 3

ignore b4533a2 Artem Alekseev Tue Aug 20 17:19:31 Add olap_workload to stability_tests (#8047)
ignore 7ab6040 zverevgeny Wed Aug 21 00:01:01 Simplify tpc-ds q34, q73 a bit to help an optimizer (#7969)
good f2788b5 ivanmorozov333 Thu Aug 22 07:58:10 multi heads allocations manager (#8126)
good c1c1c23 Semyon Thu Aug 22 10:48:52 Add unit-test for data eviction in tiering (#7971)
good bb1c803 ivanmorozov333 Thu Aug 22 13:07:13 correct portions index usage for control memory scan intervals (#8135)
good a5de48c Semyon Thu Aug 22 13:08:17 Don't skip null values on COUNT(*) in column shards (#8024)
good baf6413 Artem Alekseev Thu Aug 22 13:56:25 Add scan/insert while alter test (#8094)
good 3e3ffd1 Semyon Thu Aug 22 13:58:29 Add column shard Controller parameters to AppData (#8000)
good b4a5ff5 Alexander Avdonkin Fri Aug 23 11:51:02 Added sparsed columns tests for standalone tables (#8167)
good c249b27 Semyon Fri Aug 23 15:54:32 Forbid creating empty tiering rules in column shards (#8154)
ignore 78bb468 ildar-khisambeev Fri Aug 23 18:39:09 count-min-sketch as index (#7978)
good b5dba78 ivanmorozov333 Sat Aug 24 08:15:09 dont create temporary sparsed array on merging (#8174)
ignore d425c83 ildar-khisambeev Mon Aug 26 14:20:36 Revert add status PROCESSING to statistics proto (#8253)
ignore 1c09c5d Semyon Tue Aug 27 14:57:32 Add concurrent tiering updates to OLAP scenario test (#8262)
ignore 14d8e1b ildar-khisambeev Tue Aug 27 17:25:34 fix tests (#8323)
good f5de1a7 ivanmorozov333 Wed Aug 28 19:02:58 speed up merging for sparsed columns (#8365)
good 8d6b42b ivanmorozov333 Thu Aug 29 12:17:43 signals about long_tx writing (#8407)
good bf8c633 Semyon Thu Aug 29 17:21:13 Fix table stats aggregation for OLAP tables (#8417)
good 297d5b4 Alexander Avdonkin Fri Aug 30 09:48:56 Multicolumn sparsed test (#8284)
good 684bdae ivanmorozov333 Sun Sep 1 11:58:21 validations for prevent incorrect data in local db (#8477)
conflict f4b2c61 Semyon Sun Sep 1 19:05:25 Add unit test: drop then add column with different type (#8446)

(локи)conflict 1f29b7c (из stable-24-3-analytics) Mon Sep 9 18:48:34 [Stable 24 3 8 analytics] EvWrite & CTAS (#8861) Nikita Vasilev

(локи)conflict 1af55df ivanmorozov333 Mon Sep 2 12:26:39 tx interactions manager (#8050)
<--cs_progress_5
pacify py linter 33195d6

good 0532ed6 Artem Alekseev Mon Sep 2 13:00:37 Add registrator to TTxChainSession (#8535)
good 68720d1 ivanmorozov333 Mon Sep 2 14:54:55 Fix column livecycle (#8580)
(локи)good 5f2c357 ivanmorozov333 Mon Sep 2 14:55:23 provide correct path id to lock info (#8591)
good 892228d ivanmorozov333 Mon Sep 2 18:32:30 fix race on enqueue tx_commit in case long synchronization (#8588)
(локи)good b8d1289 ivanmorozov333 Mon Sep 2 18:32:39 remove validation throught available reask in timeout (#8609)
(локи)good 6fca8c1 ivanmorozov333 Mon Sep 2 19:27:00 we have to notify cs at first to add lock information (#8614)
good 4a4dbe2 Artem Alekseev Tue Sep 3 10:58:31 Add UpsertWhileSplitTest (#8472)
(локи)conflict b666aa2 ivanmorozov333 Wed Sep 4 09:30:05 immediate writing with no tx (#8697)
good 6180869 ivanmorozov333 Wed Sep 4 10:58:06 reduce logging rate for stats info (#8683)
good a29e93b ivanmorozov333 Wed Sep 4 11:07:56 switch on mkql factory for cs (#8682)
good 766ee48 Semyon Wed Sep 4 12:01:31 Fix partition count in column table description (#8589)
conflict 65e10f5 Semyon Wed Sep 4 12:02:58 remove blob columns not present in current scheme
good 53df48f ivanmorozov333 Wed Sep 4 12:05:24 use ss tablet id for schemeshard and sender for tx_proxy (#8684)
good 15da817 ivanmorozov333 Wed Sep 4 14:52:23 Fix broken txs (#8714)
good ba7abc4 ivanmorozov333 Wed Sep 4 15:00:26 fix case indexation removed table (#8713)
good 2fb4cac ivanmorozov333 Wed Sep 4 16:00:59 split factor increased to avoid tests interaction by timeout (#8728)
good 0ba24ba ivanmorozov333 Wed Sep 4 17:23:24 additional signals for commit timings control (#8729)
conflict f9b4a7d Artem Alekseev Wed Sep 4 18:25:23 Add EnableAlterSharding option (#8707)
conflict dc3cf10 Semyon Thu Sep 5 15:44:02 add feature flag: enable olap tiering (#8719)
good 0923723 ivanmorozov333 Thu Sep 5 18:13:34 fix import build (#8769)
good 1257b2f Semyon Fri Sep 6 15:50:18 add missing enum value to win-specific code (#8854)
ignore 3ca8730 Artem Alekseev Fri Sep 6 16:06:26 Fix Date32 json serialization for column store (#8848)

c6d4d1b Nikita Vasilev Sat Sep 7 19:39:15 2024 +0300 Olap TableStore read/write test (#8858)

conflict c0d60de ivanmorozov333 Sun Sep 8 14:04:25 remove table only for empty insert table (#8822)
good 0ab0d7a ivanmorozov333 Mon Sgiep 9 09:50:46 Fix write ids usage (#8909)
good e546733 ivanmorozov333 Mon Sep 9 10:29:25 speed up column ids checkers and finder (#8906)
ignore 68c8f3f Nikita Vasilev Mon Sep 9 11:48:12 2024 +0300 Fix locks tests (#8733)
good 2ab0fd8 Artem Alekseev Mon Sep 9 16:02:54 Add PARTITIONS_COUNT setting to yql (#7602)
good 4efd77c Alexander Avdonkin Mon Sep 9 21:48:46 Implemented limit on olap table columnt count (#8742)
good 7001202 ivanmorozov333 Tue Sep 10 12:17:22 skip long tx writing to in control locks (#8924)
conflict 26c9911 Artem Alekseev Tue Sep 10 17:25:20 Fix PARTITION_COUNT option name and add tests (#9012)
ignore 4238641 zverevgeny Tue Sep 10 19:37:02 CODEOWNERS CS (#9025)
conflict 189aa4b Vladislav Gogov Tue Sep 10 22:12:56 Add feature flag: enable olap compression (#8726)
conflict 9b1e630 ivanmorozov333 Wed Sep 11 11:17:51 Normalizer insert records (#9010)
(локи)good 618f5cf ivanmorozov333 Wed Sep 11 11:18:55 correct EvWrite locks processing (#8931)
conflict 7d436f8 Artem Alekseev Wed Sep 11 11:39:12 Add decimal tests for columnshard (#8959)

<-- 4f8fa3d cs_progress_6 tiny diff with main 7d436f8 in ydb/core/tx/columnshard/ ydb/core/formats/

ignore 71c4756 Nikita Vasilev Wed Sep 11 11:39:47 2024 +0300 HTAP: Query Processor (#8917)
ignore 99bd033 Vladislav Gogov Wed Sep 11 15:15:46 2024 +0300 Added test with disabled feature flag "EnableOlapCompression" (#9077)
good 3db740c Nikita Vasilev Wed Sep 11 15:22:43 2024 +0300 Disable CTAS by default (#9057)
good 658b27b ivanmorozov333 Thu Sep 12 12:11:11 2024 +0300 speed up logging exp (#9087)
conflict ca964ba Alexander Avdonkin Thu Sep 12 14:04:41 2024 +0300 Sparsed columns are disabled by default and can be optionally enabled (#9002)
good 8585364 ivanmorozov333 Thu Sep 12 16:37:13 2024 +0300 speed up scan initialization (#9086)
ignore 67cacf0 ivanmorozov333 Thu Sep 12 20:07:19 2024 +0300 htap locks (#9121)
ignore 8fca2df Artem Alekseev Fri Sep 13 14:19:46 2024 +0300 Fix verify in TSharingTransactionOperator::DoParse (#9158)
ignore acee834 Nikita Vasilev Fri Sep 13 15:37:03 2024 +0300 Fix sink index column order (#9190)
itnore 35e326d Nikita Vasilev Fri Sep 13 15:45:17 2024 +0300 HTAP: single arbiter (#9199)
ignore 1960e5f Nikita Vasilev Fri Sep 13 18:16:02 2024 +0300 Smaller memory footprint for shards info (#9207)
e3c9723 zverevgeny Sun Sep 15 09:20:22 2024 +0300 Turn on GCCountersNormalizer (#8166)

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions