[Feature] Support pk table migration (#25521) #25522

srlch · 2023-06-19T02:29:18Z

Support primary key table disk migration

Fixes #25521

What type of PR is this:

Checklist:

I have added test cases for my bug fix or my new feature
This pr will affect users' behaviors
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function

Bugfix cherry-pick branch check:

nshangyiming

I have reviewed the FE part, for BE part, plz ask @decster @chaoyli for review

be/src/storage/task/engine_storage_migration_task.cpp

wyb · 2023-07-31T03:12:52Z

fe/fe-core/src/main/java/com/starrocks/leader/ReportHandler.java

                db.readLock();
                try {
                    OlapTable table = (OlapTable) db.getTable(tabletMeta.getTableId());
-                    if (table.getKeysType() == KeysType.PRIMARY_KEYS) {
-                        // Currently, primary key table doesn't support tablet migration between local disks.
-                        continue;
-                    }
                } finally {
                    db.readUnlock();
                }


987-996 lines can be removed

wyb · 2023-07-31T03:14:59Z

fe/fe-core/src/main/java/com/starrocks/catalog/Table.java

@@ -631,7 +631,7 @@ public boolean needSchedule(boolean isLocalBalance) {
        }

        OlapTable olapTable = (OlapTable) this;


Suggested change

OlapTable olapTable = (OlapTable) this;

wyb · 2023-07-31T03:37:14Z

be/src/storage/tablet_manager.cpp

@@ -116,7 +117,9 @@ Status TabletManager::_add_tablet_unlocked(const TabletSharedPtr& new_tablet, bo
        auto new_version = (new_rowset == nullptr) ? -1 : new_rowset->end_version();
        old_tablet->release_header_lock();
        bool replace_old = (new_version > old_version) || (new_version == old_version && new_time > old_time);
-        if (replace_old) {
+        // In case of pk table migration, version and rowset can not be acquired by
+        // the above api. We may need to force to replace the old tablet in this case by kMoveFilesToTrash mode


use pk api to check? better to keep version and time check for safety

be/src/storage/task/engine_storage_migration_task.cpp

wanpengfei-git · 2023-08-18T07:27:28Z

[FE PR Coverage Check]

😍 pass : 63 / 72 (87.50%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/clone/TabletScheduler.java	21	27	77.78%	[1383, 1384, 1393, 1394, 1395, 1397]
🔵	com/starrocks/catalog/Replica.java	5	6	83.33%	[343]
🔵	com/starrocks/leader/ReportHandler.java	32	34	94.12%	[1003, 1039]
🔵	com/starrocks/catalog/Table.java	1	1	100.00%	[]
🔵	com/starrocks/common/Config.java	1	1	100.00%	[]
🔵	com/starrocks/server/LocalMetastore.java	1	1	100.00%	[]
🔵	com/starrocks/catalog/TabletInvertedIndex.java	2	2	100.00%	[]

be/src/storage/storage_engine.cpp

be/src/storage/task/engine_storage_migration_task.cpp

wanpengfei-git · 2023-08-21T08:46:58Z

[null Incremental Coverage Report]

😞 fail : 46 / 72 (63.89%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/clone/TabletScheduler.java	5	27	18.52%	[1363, 1364, 1365, 1366, 1367, 1369, 1370, 1371, 1374, 1375, 1377, 1379, 1382, 1383, 1384, 1387, 1389, 1390, 1393, 1394, 1395, 1397]
🔵	com/starrocks/catalog/TabletInvertedIndex.java	1	2	50.00%	[176]
🔵	com/starrocks/catalog/Replica.java	5	6	83.33%	[343]
🔵	com/starrocks/leader/ReportHandler.java	32	34	94.12%	[1003, 1039]
🔵	com/starrocks/catalog/Table.java	1	1	100.00%	[]
🔵	com/starrocks/common/Config.java	1	1	100.00%	[]
🔵	com/starrocks/server/LocalMetastore.java	1	1	100.00%	[]

Signed-off-by: srlch <linzichao@starrocks.com>

sonarqubecloud · 2023-08-22T07:38:31Z

SonarCloud Quality Gate failed.

1 Bug
0 Vulnerabilities
0 Security Hotspots
56 Code Smells

0.0% Coverage
0.0% Duplication

The version of Java (11.0.20) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more here

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

wanpengfei-git · 2023-08-22T10:07:45Z

[FE Incremental Coverage Report]

😍 pass : 63 / 72 (87.50%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/clone/TabletScheduler.java	21	27	77.78%	[1383, 1384, 1393, 1394, 1395, 1397]
🔵	com/starrocks/catalog/Replica.java	5	6	83.33%	[343]
🔵	com/starrocks/leader/ReportHandler.java	32	34	94.12%	[1004, 1040]
🔵	com/starrocks/catalog/Table.java	1	1	100.00%	[]
🔵	com/starrocks/common/Config.java	1	1	100.00%	[]
🔵	com/starrocks/server/LocalMetastore.java	1	1	100.00%	[]
🔵	com/starrocks/catalog/TabletInvertedIndex.java	2	2	100.00%	[]

wanpengfei-git · 2023-08-22T10:07:56Z

[BE Incremental Coverage Report]

😞 fail : 88 / 144 (61.11%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	src/storage/storage_engine.cpp	0	5	00.00%	[950, 951, 952, 954, 955]
🔵	src/storage/task/engine_storage_migration_task.cpp	48	93	51.61%	[117, 118, 119, 120, 129, 130, 131, 132, 235, 236, 237, 238, 239, 240, 273, 274, 281, 282, 287, 288, 289, 290, 499, 500, 501, 502, 503, 504, 505, 517, 518, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 553, 554, 555]
🔵	src/storage/tablet_updates.cpp	17	21	80.95%	[2737, 4588, 4589, 4606]
🔵	src/storage/tablet_manager.cpp	22	24	91.67%	[1527, 1528]
🔵	src/storage/snapshot_manager.cpp	1	1	100.00%	[]

Support primary key table disk migration Fixes StarRocks#25521

mergify bot assigned srlch Jun 19, 2023

srlch force-pushed the feature_support_pk_migration branch 4 times, most recently from fc2126d to 5a47004 Compare July 11, 2023 08:42

nshangyiming previously approved these changes Jul 12, 2023

View reviewed changes

nshangyiming enabled auto-merge (squash) July 12, 2023 02:37

auto-merge was automatically disabled July 12, 2023 02:42
Head branch was pushed to by a user without write access

srlch dismissed nshangyiming’s stale review via f4ef300 July 12, 2023 02:42

srlch force-pushed the feature_support_pk_migration branch from 5a47004 to f4ef300 Compare July 12, 2023 02:42

nshangyiming previously approved these changes Jul 12, 2023

View reviewed changes

srlch dismissed nshangyiming’s stale review via 31d3bf1 July 18, 2023 07:27

srlch force-pushed the feature_support_pk_migration branch 5 times, most recently from 83e3d2d to 79c49f6 Compare July 19, 2023 10:26

chaoyli reviewed Jul 27, 2023

View reviewed changes

be/src/storage/task/engine_storage_migration_task.cpp Show resolved Hide resolved

wyb reviewed Jul 31, 2023

View reviewed changes

srlch force-pushed the feature_support_pk_migration branch 6 times, most recently from cc07bf4 to 33aa5f9 Compare August 9, 2023 06:44

srlch force-pushed the feature_support_pk_migration branch 5 times, most recently from 498df63 to 0905216 Compare August 11, 2023 07:28

srlch force-pushed the feature_support_pk_migration branch from b92a0c6 to 0fa6284 Compare August 16, 2023 02:19

gengjun-git previously approved these changes Aug 16, 2023

View reviewed changes

srlch dismissed gengjun-git’s stale review via 8b4bb29 August 18, 2023 01:55

srlch force-pushed the feature_support_pk_migration branch 4 times, most recently from eb390b3 to 8df3252 Compare August 18, 2023 05:03

chaoyli reviewed Aug 18, 2023

View reviewed changes

be/src/storage/task/engine_storage_migration_task.cpp Show resolved Hide resolved

srlch force-pushed the feature_support_pk_migration branch from 8df3252 to 4c2c05b Compare August 21, 2023 06:53

srlch force-pushed the feature_support_pk_migration branch 4 times, most recently from 4196f0a to 9779913 Compare August 22, 2023 03:16

chaoyli previously approved these changes Aug 22, 2023

View reviewed changes

srlch dismissed chaoyli’s stale review via a6f22dc August 22, 2023 03:39

srlch force-pushed the feature_support_pk_migration branch 3 times, most recently from 8f78ded to 1170cba Compare August 22, 2023 06:05

[Feature] Support pk table migration (StarRocks#25521)

45b9e2d

Signed-off-by: srlch <linzichao@starrocks.com>

srlch force-pushed the feature_support_pk_migration branch from 1170cba to 45b9e2d Compare August 22, 2023 07:24

gengjun-git approved these changes Aug 22, 2023

View reviewed changes

chaoyli approved these changes Aug 22, 2023

View reviewed changes

chaoyli merged commit 1341138 into StarRocks:main Aug 22, 2023

Jay-ju pushed a commit to Jay-ju/starrocks that referenced this pull request Sep 7, 2023

[Feature] Support pk table migration (StarRocks#25521) (StarRocks#25522)

5caf661

Support primary key table disk migration Fixes StarRocks#25521

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Support pk table migration (#25521) #25522

[Feature] Support pk table migration (#25521) #25522

srlch commented Jun 19, 2023 •

edited

Loading

nshangyiming left a comment

wyb Jul 31, 2023

wyb Jul 31, 2023

srlch Aug 9, 2023

wyb Jul 31, 2023

srlch Aug 11, 2023

wanpengfei-git commented Aug 18, 2023

wanpengfei-git commented Aug 21, 2023

sonarqubecloud bot commented Aug 22, 2023

wanpengfei-git commented Aug 22, 2023

wanpengfei-git commented Aug 22, 2023

		@@ -631,7 +631,7 @@ public boolean needSchedule(boolean isLocalBalance) {
		}

		OlapTable olapTable = (OlapTable) this;

[Feature] Support pk table migration (#25521) #25522

[Feature] Support pk table migration (#25521) #25522

Conversation

srlch commented Jun 19, 2023 • edited Loading

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

nshangyiming left a comment

Choose a reason for hiding this comment

wyb Jul 31, 2023

Choose a reason for hiding this comment

wyb Jul 31, 2023

Choose a reason for hiding this comment

srlch Aug 9, 2023

Choose a reason for hiding this comment

wyb Jul 31, 2023

Choose a reason for hiding this comment

srlch Aug 11, 2023

Choose a reason for hiding this comment

wanpengfei-git commented Aug 18, 2023

[FE PR Coverage Check]

file detail

wanpengfei-git commented Aug 21, 2023

[null Incremental Coverage Report]

file detail

sonarqubecloud bot commented Aug 22, 2023

wanpengfei-git commented Aug 22, 2023

[FE Incremental Coverage Report]

file detail

wanpengfei-git commented Aug 22, 2023

[BE Incremental Coverage Report]

file detail

srlch commented Jun 19, 2023 •

edited

Loading